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Non-Lethal  Weapons  Effectiveness  Assessment 
Development  and  Verification  Study 
(RTO-TR-SAS-060) 

Executive  Summary 

During  the  1999  Washington  summit,  the  North  Atlantic  Council  (NAC)  identified  Non-Lethal  Weapons 
(NLW)  as  a  “critical,  additional  capability  needed  in  order  to  meet  the  demands  of  future  operations.” 
Recent  and  ongoing  operations  have  confirmed  this  need.  Lessons  from  peace  support  and  anti-/counter- 
terrorism  operations  have  highlighted  the  need  to  accomplish  tasks  while  minimizing  undesired  or 
collateral  effects,  which  is  key  to  the  very  nature  and  definition  of  non-lethal  weapons1: 

Non-Lethal  Weapons  are  weapons  which  are  explicitly  designed  and  developed  to  incapacitate 
or  repel  personnel,  with  a  low  probability  of  fatality  or  permanent  injury,  or  to  disable 
equipment,  with  minimal  undesired  damage  or  impact  on  the  environment. 

Although  the  need  for  NLW  has  been  formally  recognized,  NATO  and  its  member  nations  still  must 
determine  which  capabilities  to  develop  and  field  and  how  best  to  employ  them.  Understanding  NLW 
effectiveness  is  vital  to  supporting  such  decisions.  SAS-060’s  purpose  was  to  develop  a  common,  agreed 
approach  for  assessing  effectiveness.  In  line  with  the  study’s  specific  objectives,  SAS-060  formed  three 
working  groups  that: 

•  Collected  and  applied  NLW  data  to  test  the  system  effectiveness  methodology  developed  by 
SAS-060’s  predecessor  study  (SAS-035); 

•  Extended  the  methodology  to  account  for  aggregation  of  results,  at  a  point  in  time  and  over 
multiple  scenario  phases,  and  variability /uncertainty;  and 

•  Enhanced  NLW  awareness  and  integrated  with  other  NATO  efforts. 

Conclusions  -  SAS-060  partly  verified  the  system  effectiveness  methodology  (with  some  issues  remaining 
but  confidence  they  can  be  resolved),  extended  the  methodology  (with  multiple  aggregation  methods 
identified  and  implemented)  to  include  assessments  across  phases  of  a  scenario,  and  increased  NLW 
awareness. 

Recommendations  -  The  new  NATO  NLW  Roadmap  calls  for  a  SAS-060  Follow-On  Study.  SAS-060 
discussed  the  merits  of  the  recommendation.  Continued  work  on  the  effectiveness  methodology  was  not 
viewed  as  the  best  course  -  as  the  work  has  already  progressed  beyond  the  availability  of  data  to  support 
verification/validation.  SAS-060’s  two  highest  priority  topics  for  study  are: 

•  Capabilities  Assessment  -  Addressing  NATO  operational  requirements,  gaps  between  capabilities 
and  requirements,  and  wargaming/cost-benefit  analyses  of  candidate  solutions  to  gaps;  and 

•  Experimentation  -  Developing  NATO-common  protocols  for  NLW  experiments,  conducting 
experiments  to  generate  data,  and  addressing  data  shortfalls  identified  in  previous  studies. 

In  October  2007,  the  Conference  of  National  Armaments  Directors  (CNAD)  approved  a  NLW  Defence 
Against  Terrorism  (DAT)  initiative,  which  may  also  require  follow-on  support. 


1  NATO  Policy  for  Non-Lethal  Weapons,  27  September  1999. 
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Synthese 

Lors  du  Sommet  de  Washington  de  1999,  le  Conseil  de  l’Atlantique  Nord  (NAC)  a  identifie  les  armes  non 
letales  (NLW)  comme  «  une  capacite  critique  supplementaire  indispensable  pour  satisfaire  aux  exigences 
des  futures  operations  ».  Les  operations  recentes  ou  en  cours  ont  confirme  ce  besoin.  Les  leqons  tirees  des 
operations  anti-terroristes  et  de  soutien  de  la  paix  ont  fait  apparaitre  clairement  la  necessite  d’accomplir  les 
missions  tout  en  limitant  les  effets  non  desires  ou  collateraux,  ce  qui  est  indissociable  de  la  definition  et  de 
la  nature  meme  des  armes  non  letales2 : 

Les  armes  non  letales  sont  des  armes  discriminantes  qui  sont  explicitement  mises  au  point  et 
principal ement  utilisees  pour  frapper  d’ incapacity  le  personnel  et  le  materiel,  avec  un 
minimum  de  risque  mortel,  de  lesions  permanentes  au  personnel  et  de  dommages  indesirables 
aux  biens  et  a  I’environnement. 

Bien  que  la  necessite  des  NLW  ait  ete  officiellement  reconnue,  l’OTAN  et  ses  nations  membres  doivent 
encore  determiner  quelles  capacites  developper  et  employer  sur  le  terrain  et  comment  les  utiliser  de  maniere 
optimale.  La  comprehension  de  l’efficacite  des  NLW  est  cruciale  lors  de  telles  decisions.  Le  but  du 
SAS-060  etait  done  de  developper  une  approche  commune  et  convergente  devaluation  de  cette  efficacite. 
Conformement  aux  objectifs  specifiques  de  T  etude,  le  SAS-060  a  constitue  trois  groupes  de  travail,  qui : 

•  Ont  recueilli  et  applique  les  donnees  relatives  aux  NLW  en  vue  de  tester  la  methodologie 
d’efficacite  du  systeme  developpee  lors  de  l’etude  du  SAS-035,  predecesseur  du  SAS-060  ; 

•  Ont  etendu  la  methodologie  afin  qu’elle  integre  l’agregation  des  resultats,  a  un  moment  precis 
dans  le  temps  et  lors  de  multiples  phases  de  scenarios,  ainsi  que  la  variabilite  et  1’ incertitude  ;  et 

•  Ont  renforce  la  sensibilisation  aux  NLW  et  l’ont  ajoute  aux  autres  efforts  de  l’OTAN. 

Conclusions  -  Le  SAS-060  a  partiellement  verifie  la  methodologie  d’efficacite  du  systeme  (certains 
problemes  demeurent,  mais  il  existe  un  bon  espoir  de  les  resoudre),  etendu  cette  methodologie 
(par  T identification  et  la  mise  en  oeuvre  de  plusieurs  methodes  d’agregation)  pour  inclure  des  evaluations 
lors  des  differentes  phases  d’un  scenario,  et  ameliore  la  sensibilisation  aux  NLW. 

Recommandations  -  La  nouvelle  feuille  de  route  de  TOT  AN  relative  aux  NLW  prevoit  une  etude  de 
suivi  du  SAS-060.  Le  SAS-060  a  examine  le  bien-fonde  de  cette  recommandation.  Un  travail  continu  sur 
la  methodologie  d’efficacite  n’est  pas  considere  comme  etant  la  meilleure  option  -  dans  la  mesure  ou  les 
travaux  ont  deja  progresse  au-dela  de  la  disponibilite  de  donnees  venant  a  l’appui  de  la  verification  et  de  la 
validation.  Les  deux  sujets  d’etude  prioritaires  du  SAS-060  sont : 

•  L’evaluation  des  capacites  -  Determiner  les  exigences  operationnelles  de  TOT  AN,  les  ecarts 
existants  entre  les  capacites  et  les  exigences,  et  realiser  des  jeux  de  strategic  et  des  analyses  de 
rendement  des  solutions  envisagees  pour  combler  ces  ecarts  ;  et 


2 

“  Politique  de  I’OTAN  sur  les  armes  non  letales,  27  septembre  1999. 
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•  Experimentation  -  Developper  des  protocoles  communs  a  l’OT AN  pour  les  experiences  relatives 
aux  NLW,  realiser  des  experiences  en  vue  de  generer  des  donnees,  et  tenter  de  combler  le  manque 
de  donnees  identifie  dans  les  etudes  precedentes. 

En  octobre  2007,  la  Conference  des  Directeurs  Nationaux  de  l’Armement  (CDNA)  a  approuve  une  initiative 
de  defense  contre  le  terrorisme  (DAT)  par  NLW,  qui  pourrait  egalement  necessiter  soutien  et  suivi. 
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1.1  STUDY  CONTEXT 

The  challenges  presently  confronting  NATO  are  very  different  from  those  that  prevailed  throughout  the 
Cold  War.  As  demonstrated  in  ongoing  peace  support  and  anti-terrorism/counter-terrorism  operations, 
NATO  is  facing  new  types  of  threats,  undertaking  new  kinds  of  missions,  performing  a  broad  range  of 
military  tasks  within  difficult  operating  environments,  and  conducting  these  tasks  subject  to  a  variety  of 
policy  and  operational  constraints.  Lessons  learned  during  these  operations  have  highlighted  the  need  for 
non-lethal  weapons  (NLW).  The  need  to  accomplish  tasks  while  minimizing  undesired  or  collateral  effects 
speaks  to  the  very  nature  and  definition  of  non-lethal  weapons1: 

Non-Lethal  Weapons  are  weapons  which  are  explicitly  designed  and  developed  to  incapacitate 
or  repel  personnel,  with  a  low  probability  of  fatality  or  permanent  injury,  or  to  disable 
equipment,  with  minimal  undesired  damage  or  impact  on  the  environment. 

Table  1-1  summarizes  several  challenges  identified  in  recent  and  ongoing  operations  and  associated 
implications  (and  opportunities)  for  NLW. 

Table  1-1 :  Lessons  Learned  and  the  Need  for  NLW 


Challenge 

Implication/Opportunity  for  NLW 

Minimize  undesired  casualties: 

-  When  uncertain  if  a  target  is  hostile. 

-  When  others  (non-combatants  or 
friendly  forces)  are  near  a  target. 

-  When  the  desired  target  effect/  response 
is  temporary  (i.e.,  not  desired  dead  or 
destroyed). 

Offer  reversible  means  to  act  that  will: 

-  Help  discern  intent  (also  allowing  for 
earlier  use  of  lethal  force  upon 
determining  hostile  intent). 

-  Dissuade,  delay,  or  defeat  the  target. 

-  Isolate  threats  from  human  shields. 

Minimize  undesired  collateral  effects: 

-  On  infrastructure  and  facilities 
(especially  if  civilian/dual-use  or  of 
symbolic  importance). 

-  Which  have  an  environmental  impact. 

Provide  limited  duration  effects  that: 

-  Deny  adversary’ s  use  or  access  while 
preserving  for  later  use. 

-  Minimise  re-construction  needs. 

Act  when  lethal  force  may  not  be  allowed  or 
may  be  counter-productive: 

-  Policy,  treaty/legal,  or  operational 
constraints. 

-  Potential  2nd  and  3ld  order  effects. 

Tailor  effects  (type  and  magnitude)  to: 

-  Accomplish  task  requirements. 

-  Satisfy  constraints. 

-  Avoid  undesired  consequences 
(negative  perceptions  or  reactions). 

NATO  has  clearly  recognized  the  potential  offered  by  non-lethal  weapons.  During  the  1999  Washington 
Summit,  the  North  Atlantic  Council  identified  NLW  as  a  “critical,  additional  capability  needed  in  order  to 
meet  the  demands  of  future  operations”  and  established  NLW  as  a  Defence  Capabilities  Initiative  item 
(Item  EE  2(i)).  Since  then,  NLW  needs  have  been  reinforced  in  peace  support  operations  (as  highlighted  in 


1  NATO  Policy  for  Non-Lethal  Weapons,  27  September  1999. 
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conclusions  from  SAS-041/SAS-048)  and  in  an t i - tc ito r i s m/cou n te r- 1 e rro ri s m  efforts  (as  discussed  in 
MC472).  In  order  to  meet  these  needs,  NATO  has  conducted  a  variety  of  activities.  Past  work  includes 
development  and  approval  of  a  NATO  NLW  Policy,  development  of  an  initial  NATO  NLW  Roadmap, 
multiple  studies  conducted  under  the  auspices  of  the  Research  and  Technology  Organisation* 2, 
standardization  efforts  by  a  Team  of  Experts  within  a  NATO  Army  Armaments  Group  (NAAG)  Land 
Group,  an  Advanced  Research  Workshop,  a  NLW  Exhibition  sponsored  by  the  Conference  of  National 
Armaments  Directors  (CNAD),  and  a  NAAG  Quick  Reaction  Team  (QRT)  which  has  just  developed  a  new 
NATO  NLW  Roadmap. 

Although  the  need  for  NLW  has  been  formally  recognized  and  supporting  activities  begun,  NATO  and  its 
member  nations  still  must  determine  which  non-lethal  capabilities  to  develop  and  field  and  how  best  to 
employ  them.  Understanding  NLW  effectiveness  is  vital  to  supporting  such  decisions  and  to  ensuring  that 
commanders  and  troops  in  the  field  will  be  comfortable  employing  these  capabilities. 


1.2  SAS-060’S  PURPOSE 

SAS-060’s  predecessor  study  (SAS-035)  developed  a  proposed  framework  for  measuring  NLW  system 
effectiveness3,  and  the  SAS  Panel  chartered  SAS-060  to  verify  and  extend  the  SAS-035  methodology, 
with  the  following  objectives,  as  identified  in  the  study’s  Terms  of  Reference  (included  as  Annex  A): 

1)  Verify  the  Measures  of  System  Effectiveness  (MoSE)  Methodology: 

•  Identify  necessary  data  types  and  experiments  or  experimental  guidelines  for  generating  them. 

•  Make  explicit  the  process  for  describing  required  responses. 

•  Verify  the  methodology,  with  realistic  data  for  actual  system(s)  in  an  operational  context,  and 
provide  worked  example(s). 

2)  Extend  the  MoSE  Methodology: 

•  Address  potential  uses  and  associated  methods  of  aggregation. 

•  Address  confidence  levels,  including  an  examination  of  potential  sources  of  variation  and  how 
valuations  propagate  through  the  methodology’s  calculations. 

3)  Explore  development  of  Measures  of  Operational  Effectiveness  (MoOEs): 

•  Assess  the  feasibility  of  extending  the  methodology  to  account  for  operational  effectiveness 
(i.e.,  the  simultaneous  and/or  sequential  use  of  one  or  more  systems  -  non-lethal  and/or  lethal 
-  to  achieve  a  desired  outcome). 

•  Develop  and  explore  concepts  for  assessing  operational  effectiveness. 

4)  Enhance  awareness  of  NLW  work  and  integrate  with  other  NATO  efforts: 

•  Engage  with  organisations  identified  in  the  NATO  NLW  Roadmap. 

•  Conduct  briefings  for  Allied  Command-Operations  and  Allied  Command-Transformation  and 
others  as  appropriate  and  develop  and  conduct  a  seminar/workshop  if  warranted. 


Studies  include  SAS-035  (NLW  Effectiveness  Assessment,  the  predecessor  of  SAS-060),  SAS-040  (Long  Term  Scientific 

Study),  and  HFM-073  (Human  Effects). 

3  Non-Lethal  Weapons  Effectiveness  Assessment  (SAS-035  RTO-TR-085),  published  October  2004. 
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1.3  STUDY  TEAM 

SAS-060  included  participants  from  12  nations  (eleven  alliance  members  and  one  partner  nation): 

•  Belgium 

•  Canada 

•  Denmark 

•  France 

•  Germany 

•  Italy 

•  Netherlands 

•  Norway 

•  Spain 

•  Sweden 

•  United  Kingdom 

•  United  States 

In  addition  SAS-060  conducted  significant  information  exchange  with  Allied  Command  Transformation 
(ACT),  including  holding  a  SAS-060  Study  Team  meeting  at  ACT,  which  included  an  exchange  of  briefings 
and  information  materials. 

A  listing  of  individual  SAS-060  participants  is  provided  in  Annex  B. 


1.4  WORKING  GROUPS 

The  SAS-060  Study  Team  met  twice  per  year  to  oversee  and  integrate  study  efforts. 

The  Study  Team  formed  three  working  groups  to  carry  out  the  detailed  work  of  the  study.  Each  working 
group  met  as  needed  to  carry  out  assigned  responsibilities.  The  Task  1  working  group  was  responsible  for 
objective  1  from  the  Terms  of  Reference,  the  Task  2  working  group  for  objectives  2  and  3,  and  the  Task  3 
working  group  for  objective  4,  with  an  overview  of  work  packages  as  follows: 

•  Task  1  -  Methodology  Verification  and  Refinement,  which  focused  on  testing  and  refining  the 
SAS-035  system  effectiveness  methodology  with  data  from  a  variety  of  NLW  technologies 
(vehicle  arrestor  nets,  optical  disrupters,  high-power  microwaves,  kinetic  devices,  and  flash- 
bangs). 

•  Task  2  -  Aggregating  Results  for  System  and  Operational  Effectiveness,  which  examined  a 
range  of  scenarios  and  addressed  aggregation  of  results  at  a  point  in  time  (system  effectiveness) 
or  across  phases  of  a  scenario  (operational  effectiveness). 

•  Task  3  -  Enhancing  Awareness  and  Coordinating  with  Other  NATO  NLW  Efforts,  which 
developed  presentation  materials  (including  for  the  SAS-060  display  at  the  CNAD  NLW 
Exhibition),  conducted  liaison  with  other  NATO  efforts  such  as  ACT  and  the  NAAG  Quick 
Reaction  Team,  and  developed  a  NLW  Workshop  for  NATO  Defence  College. 
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1.5  REPORT  ORGANISATION 


The  main  body  of  the  report  includes  an  overview  of  the  three  tasks  and  results  developed  by  their 
respective  working  groups,  followed  by  conclusions  and  recommendations: 

•  Chapter  2  describes  the  efforts  of  the  Task  1  working  group,  presenting  the  approach  and  results 
for  testing  and  refining  the  NLW  system  effectiveness  methodology. 

•  Chapter  3  describes  the  efforts  of  the  Task  2  working  group  regarding  aggregation  of  results  and 
implications  for  system  and  operational  effectiveness. 

•  Chapter  4  describes  the  efforts  of  the  Task  3  working  group  to  enhance  NLW  awareness  and 
coordinate  with  other  NATO  efforts. 

•  Chapter  5  presents  conclusions  and  recommendations,  including  SAS-060’s  proposed  response 
to  the  NAAG  QRT  recommendation  for  a  SAS-060  Follow-On  Study. 

A  series  of  annexes  provide  additional  detail. 
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2.1  INTRODUCTION 

In  2004,  NATO’s  Research  and  Technology  Organisation  published  the  final  report  of  SAS-0351.  This  report 
proposed  an  effectiveness  methodology,  starting  with  physical  weapon  characteristics  and  calculating  in  turn 
Measures  of  Performance  (MoP),  Measures  of  [target]  Response  (MoR),  Measures  of  System  Effectiveness 
(MoSE),  and  finally  Measures  of  Operational  Effectiveness  (MoOE). 

2.1.1  Objectives 

The  aim  of  Task  1,  as  stated  in  the  SAS-060  Terms  of  Reference2,  was  to  verify  the  Measures  of  System 
Effectiveness  (MoSE)  methodology: 

•  Identifying  necessary  data  types  and  experiments  or  experimental  guidelines  for  generating  them; 

•  Making  explicit  the  process  for  generating  required  responses;  and 

•  Verifying  the  methodology,  with  realistic  data  for  actual  system(s)  in  an  operational  context,  and 
providing  worked  examples. 

2.1.2  Work  Packages 

The  Task  1  working  group  investigated  the  following  NLW  technologies  and  specific  weapon  systems: 

1)  Vehicle  Arrestor  Nets 

A  vehicle  arrestor  net,  placed  on  the  roadway,  will  bring  a  vehicle  to  a  halt  rapidly  but  without 
injury  to  occupants.  Vehicle  arrestor  nets  are  in  use  by  several  armed  forces,  and  they  have  been 
deployed  operationally.  Extensive  trials  data  is  available,  in  terms  of  stopping  distances  against  a 
wide  range  of  vehicle  types  under  varied  environmental  conditions,  and  some  data  is  available  on 
performance  in  operational  use. 

2)  Optical  Disrupter 

An  optical  disrupter  shines  a  light  beam  or  flash  into  the  eye,  at  an  intensity  which  will  cause 
disorientation  to  the  subject  but  with  no  risk  of  permanent  damage  to  the  eye.  Based  on  a  conceptual 
design,  experimental  data  was  postulated. 

3)  High  Power  Microwave  (HPM)  Weapon 

A  HPM  weapon  uses  a  non-nuclear  electromagnetic  pulse  (NNEMP)  to  cause  temporary  or 
permanent  damage  to  electronic  equipment.  No  weapon  of  this  type  is  currently  commercially 
available,  but  experimental  data  on  potential  effectiveness  of  such  a  weapon  is  available. 

4)  Kinetic  Energy  Weapons 

These  are  weapons  which  throw  a  projectile,  such  as  a  blini  or  baton  round,  the  impact  of  which  will 
temporarily  incapacitate  a  person  without  causing  permanent  damage.  Experimental  performance, 
range,  accuracy  and  impact  data  was  available  for  several  kinetic  energy  weapons. 


1  NATO  RTO  Technical  Report  TR-085,  AC/323(SAS-035)TP/44  dated  October  2004. 

2 

See  the  Terms  of  Reference  and  Technical  Activity  Proposal  Activity  Ref  RTG-020  dated  August  2004. 
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5)  Flash-Bangs 

Flash-bangs  are  pyrotechnic  devices,  typically  in  the  form  of  grenades,  which  produce  a  loud  bang 
and  brilliant  flash;  they  incapacitate  people  within  some  radius  of  the  burst  point,  without 
permanent  damage  to  sight  or  hearing.  Experiments  have  shown  the  effect  of  flash-bangs  on  the 
ability  of  subjects  to  carry  out  simple  tasks  (such  as  aiming  weapons). 

Table  2-1  presents  a  summary  of  NLW  systems  and  types  of  data  available. 


Table  2-1 :  Summary  of  NLW  Assessments 


Technology 

Data  Status 

Vehicle  Arrestor 

Extensive  experimental  data  on  stopping  distances  (MoR) 

Some  operational  data 

Optical  Disrupter 

Physical  characteristics  (PWC)  and  some  experimental 
data  (MoP) 

High-Power  Microwave 

Experimental  results  on  performance  and  effect  on 
electronic  hardware  (MoP/MoR) 

Kinetic  Energy  Weapons 

Experimental  results  on  performance  (MoP) 

Flash-Bangs 

Results  of  experiments  on  human  subjects  (MoP/MoR) 

2.2  APPROACH 

The  approach  involved  working  through  the  steps  of  the  SAS-035  methodology  and  evaluating  MoSEs 
for  a  range  of  NLW  across  military  tasks  drawn  from  a  common  scenario.  A  detailed  description  of  the 
SAS-035  methodology  is  given  in  Section  3  of  the  SAS-035  Final  Report.  Figure  2-1  provides  an  overview. 
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Figure  2-1 :  The  SAS-035  Methodology. 

The  methodology  consists  of  the  following  steps: 

1)  A  scenario  is  defined  in  terms  of  an  operational  situation  and  military  tasks. 

2)  Physical  weapon  characteristics  (PWCs)  -  such  as  weight,  calibre,  muzzle  velocity,  and  power 
requirements  of  the  weapon(s)  -  are  determined. 

3)  Measures  of  Performance  are  derived  from  PWCs  and  environmental  factors,  with  typical  MoPs 
including: 

•  Terminal  momentum  of  a  baton  round; 

•  Optical  intensity  of  a  flash-bang; 

•  Area  coverage  of  a  net;  and 

•  Field  strength  of  a  directed  energy  weapon. 

4)  Required  Responses  (RRs)  specify  three  time  profiles: 

•  Task  accomplishment  (effects  on  intended  targets); 

•  Target  constraint  satisfaction  (effect  reversibility  and  target  recovery);  and 

•  Collateral  constraint  satisfaction  (reversibility/recovery  for  own  forces,  non-combatants, 
infrastructure,  etc.). 

RRs  are  expressed  in  terms  of  seven  Basis  Response  (BR)  variables:  Mobility,  Communication, 
Physical  Function,  Sense  and  Interpret,  Group  Cohesion,  Motivation,  and  Identification. 
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5)  MoPs,  together  with  Target  Response  Characteristics,  are  then  used  to  estimate  the  actual  target 
response.  These  Measures  of  Response  (MoRs)  are  time  profiles  of  the  same  Basis  Response  as  used 
for  the  RRs. 

6)  RR  vs.  MoR  time  profiles  are  graphically  compared  to  calculate  the  degree  to  which  the  measured 
response  meets  the  required  response  in  terms  of: 


•  Target  constraint  satisfaction  (designated  P2) 


•  Collateral  constraint  satisfaction  (designated  P3) 


These  comparisons  are  made  for  each  of  the  7  Basis  Response  variables,  to  give  21 
Measures  of  System  Effectiveness  (MoSE)  elements;  more  details  on  the  comparison 
process  are  given  in  Annex  C. 
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7)  The  MoSE  are  then  fed  into  a  model  of  the  scenario,  and  compared  to  a  measure  of  Required 
Outcome  to  arrive  at  a  Measure  of  Operational  Effectiveness.  However,  this  step  in  the  process  is 
outside  the  scope  of  Task  1. 

The  aim  of  Task  1,  therefore,  was  to  verify  the  practicability  of  carrying  out  Steps  1)  through  6)  for  real 
weapon  systems. 


2.3  SCENARIO 

The  scenario  used  for  the  Task  1  verification  exercise  involved  protection  of  a  Brigade  and  Battalion  HQ, 
depicted  in  Figure  2-2,  in  a  UN  Peace  Enforcement  operation.  The  overall  mission  required  restoring  law 
and  order  in  the  area  of  responsibility  and  providing  general  area  security.  The  specific  military  objective 
was  to  maintain  security  of  the  camp  and  freedom  of  movement  of  own  troops. 


Main  Road 


N 

no  trespassing  zone 
fence 


10-20  m 


emergency  exit 
watch-tower 


Side  Road  Check  Point 


Figure  2-2:  Scenario  Layout. 


Two  possible  threats  to  the  HQ  were  considered: 

•  A  low  intensity  threat  involving  a  local  population  wanting  to  break  into  the  camp  to  steal  material 
and  food  supplies;  and 

•  A  higher  intensity  threat  (considered  only  for  the  HPM  Weapon),  with  the  scenario  extended  to 
include  organised  warring  factions  equipped  with  mortars,  grenades,  and  automatic  weapons,  with 
a  command  post  1  km  from  the  camp. 

The  scenario  was  divided  into  four  phases  corresponding  to  the  progress  of  an  attack  on  the  HQ  compound, 
with  military  tasks  identified  for  each  phase.  The  MoSEs,  for  each  of  the  weapon  systems  considered  in  Task 
1,  were  then  evaluated  in  the  context  of  the  military  tasks  associated  with  defeating  these  possible  threats. 

A  more  detailed  description  of  the  scenario  is  given  in  Annex  C. 
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2.4  VEHICLE  STOPPER  EXAMPLE 

This  example  made  use  of  data  available  on  an  already  fielded  device  that  can  rapidly  be  deployed  to  arrest  a 
suspect  vehicle  in  a  controlled,  non-lethal  manner.  In  applying  the  methodology,  the  Task  1  assessment 
included  the  following  assumptions: 

NLW  Characteristics  and  Employment  Considerations 

•  Sufficient  resources  are  available  at  all  times  to  deploy  the  vehicle  stopper  on  the  inside  of  the 
northern  checkpoint  covering  the  whole  width  of  the  road. 

•  Three  seconds  have  been  allowed  for  the  time  from  the  decision  to  stop  the  vehicle  to  the 
completion  of  system  deployment. 

Target  ( Approaching  Vehicle)  Characteristics 

•  The  vehicle  is  not  able  to  swerve  around  the  vehicle  stopper,  and  vehicles  aim  to  have  a 
similar  speed  on  both  sides  of  the  checkpoint. 

•  In  this  analysis,  non-intended  targets  are  our  vehicles. 

•  In  line  with  trials  data,  the  driver  will  keep  his  foot  on  the  accelerator,  even  after  the  system 
has  engaged  the  front  tyres,  until  the  vehicle  is  stopped. 

Weapon-Target  Interactions 

•  The  system  will  not  stop  a  vehicle  travelling  above  1 10  kph  (70  mph). 

•  Due  to  the  puncturing  of  target  vehicles’  tyres,  ten  minutes  has  been  allowed  for  the  vehicle  to 
start  moving  again,  and  it  is  assumed  those  vehicles  are  only  capable  of  reaching  half  their 
initial  speed  after  being  cut  free  (the  70  mph  vehicle  remains  at  the  same  speed,  as  this  vehicle 
stopper  does  not  affect  it). 

•  Knock  on  effects  of  deploying  the  vehicle  stopper  have  not  been  examined. 

•  Sample  trials  data  has  been  used,  and  it  is  assumed  the  speeds  assessed  within  the  trials  data 
are  achievable  in  the  scenario. 

•  Two  minutes  has  been  allowed  to  detain  the  occupants  of  the  stopped  target  vehicle. 

2.4.1  Scenario  Factors 

Given  the  scenario,  the  vehicle  stopper  will  be  deployed  in  the  following  military  tasks: 

•  Phase  2  -  Task  1:  Stop  or  divert  unauthorised  personnel  and  vehicles  from  reaching  the  perimeter 
fence  and  main  entrance. 

•  Phase  2  -  Task  7:  Stop  unauthorised  vehicles  from  reaching  the  camp  main  entrance  and  apprehend 
occupants. 

•  Phase  4  -  Task  8:  Prevent  enemy  from  escaping. 

Due  to  the  similarity  of  the  above  tasks,  the  rest  of  this  assessment  focuses  on  the  first  task,  with  the 
results  assumed  to  be  applicable  to  the  other  tasks. 

The  mode  of  deployment  is: 

•  The  Military  Commander  sees  an  approaching  target  vehicle  and  gives  the  signal  to  deploy  the 
vehicle  stopper  (which  takes  3  seconds  for  operators  to  deploy). 
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•  The  system  engages  the  vehicle  which  should  ideally  come  to  a  standstill  50  metres  away  from  the 
cut  off  team  standing  ready  to  apprehend  occupants. 

2.4.2  Basis  Response  (BR)  Variables 

The  basis  response  variables  to  which  the  vehicle  stopper  contributes  are: 

•  Mobility:  The  system  arrests  vehicles,  bringing  them  to  a  controlled  stop.  In  trials  it  has  successfully 
stopped  Front  Wheel  Drive  (FWD),  Real'  Wheel  Drive  (RWD),  Four  Wheel  Drive  (4WD)  and 
vehicles  equipped  with  run-flat  tyres.  The  target  response  will  be  measured  by  reduction  in  speed 
over  time. 

•  Physical  Function:  The  impact  on  Physical  Function  is  due  to  the  NLW’s  ability  to  ensure  vehicles 
work  at  reduced  efficiency  -  for  example,  the  system  could  immobilise  a  vehicle,  allowing  the  arrest 
of  the  driver  or  passenger. 

The  vehicle  stopper  is  not  regarded  as  having  any  effect  on  the  other  Basis  Response  variables. 

2.4.3  Required  Response  and  MoR 

It  is  assumed  that  the  Military  Commander  wants  the  vehicle  to  come  to  a  standstill  as  close  to  his  cut  off 

team  (50  metres  away)  as  possible.  The  RR  curve  is  as  shown  in  Figure  2-3. 
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Figure  2-3:  Required  Response  (RR)  Curve. 


To  examine  the  flexibility  of  the  methodology,  three  cases  were  assessed: 

1)  A  vehicle  at  30  mph  (13.41  m/s); 

2)  A  vehicle  at  49  mph  (21.90  m/s);  and 

3)  A  vehicle  at  70  mph  (31.29  m/s),  which  is  not  slowed  at  all  by  this  vehicle  stopper. 

The  MoR  curves  for  these  cases,  derived  from  trials  data  are  shown  in  Figure  2-4  -  Figure  2-6,  which  also 
show  the  RR  curve  for  comparison. 
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Figure  2-6:  MoR  for  70  mph  Case. 


2.4.4  Measures  of  System  Effectiveness  (MoSE) 

Based  on  these  results,  and  using  the  method  described  in  Annex  C,  values  for  Pi,  P2  and  P3  were  derived 
for  the  two  Basis  Responses  -  Mobility  and  Physical  Function;  because  the  same  quantity  was  used  to 
measure  both  Responses  the  results  are  the  same.  The  results  are  shown  in  Table  2-2;  the  values  for  P3  are 
based  on  the  assumption  that  no  non-target  vehicles  would  be  affected  by  the  vehicle  stopper  in  this 
scenario. 


Table  2-2:  MoSE  Results 


Case 

Pi 

P2 

P3 

30  mph 

0.9959 

Tends  to  0.5 

1 

49  mph 

0.9748 

Tends  to  0.5 

1 

70  mph 

0 

1 

1 

2.4.5  Sensitivity  Analyses 

The  following  sensitivity  cases  were  examined: 

•  A  required  response  with  a  constant  rate  of  deceleration. 

•  Varying  the  distance  between  the  vehicle  stopper  and  the  cut  off  team  with  a  required  response 
that  is  based  on: 

•  An  instantaneous  reduction  in  speed  to  zero. 

•  A  constant  rate  of  deceleration. 

Pi  values  were  calculated  for  the  above  for  both  the  30  mph  and  49  mph  cases.  Table  2-3  compares  the 
results  for  the  constant  deceleration  case  with  the  base  case. 
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Case 
30  mph 
49  mph 


Table  2-3:  Comparison  of  MoSEs  for  Alternative  Required  Responses 

Instantaneous  P,  Constant  Deceleration  Pi 


0.9959 


0.9899 


0.9748 


0.9677 


Pi’s  variation  with  stopping  distance  for  the  two  cases  is  shown  in  Figure  2-7  and  Figure  2-8.  Figure  2-7 
shows  results  for  varying  distances  between  vehicle  stopper  deployment  and  the  cut-off  team,  based  on  the 
required  response  for  the  vehicle  stopping  instantaneously.  Figure  2-8  shows  similar  results  for  constant 
deceleration. 


— ♦ — 30mph  ■  49m  ph 


Figure  2-7:  Pi  Values  (Instantaneous  Stopping). 


-• — 30mph  ■  49m  ph 


Figure  2-8:  Pi  Values  (Constant  Deceleration). 
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2.4.6  Summary  from  Vehicle  Stopper  Assessment 

The  Task  1  working  group  concluded  that: 

•  The  proposed  effectiveness  methodology  can  be  successfully  applied  (and  defining  a  realistic 
Required  Response  is  key  to  this  success). 

•  The  methodology  has  been  proved  to  be  sensitive  to  changes  in  inputs. 

Issues  identified  in  the  course  of  this  exercise  were: 

•  Several  assumptions  were  required  to  limit  the  scope  of  work  so  the  methodology  could  be  applied. 

•  Calculation  of  Pi  values  is  clear;  P2  and  P3  required  some  interpretation,  and  a  cut-off  time  was 
used  otherwise  graph  areas  were  unbounded.  (This  issue  was  also  identified  in  the  other  NLW 
assessments,  and  proposed  refinements  to  the  methodology  -  revising  the  P2  calculation  and 
eliminating  P3  -  are  discussed  later  in  this  chapter.) 

•  Required  Responses  are  based  on  Military  stakeholder  opinion,  which  might  vary  between 
individuals. 

•  Further  issues  might  be  identified  when  comparing  multiple  NLW  options. 


2.5  FLASH-BANG  EXAMPLE 

The  military  tasks  (please  see  Annex  B  for  more  details)  in  which  it  was  assumed  that  flash-bangs  could 
be  applied  were: 

Phase  2  -  Task  2:  Disperse  personnel  or  vehicles  in  the  No  Trespass  Zone 
Phase  2  -  Task  4:  Disorientate  personnel 

Phase  3  -  Task  2:  Disperse  personnel  or  vehicles  in  the  No  Trespass  Zone  or  Side  Road 

Phase  3  -  Task  4:  Disorientate  personnel 

Phase  4  -  Task  2:  Incapacitate  vehicles  and  individuals 

The  available  data  is  not  applicable  to  the  tasks  in  which  a  group  of  people  is  dispersed  by  means  of  a 
flash-bang  device.  For  the  purpose  of  this  investigation  the  decision  was  made  to  take  Phase  2  -  Task  4 
and  Phase  3  -  Task  4  as  starting  points.  The  difference  between  the  selected  tasks  is  found  in  the 
associated  constraints.  During  Phase  2  the  actions  against  individuals  must  have  no  permanent  effect, 
whereas  Phase  3  permits  for  some  permanent  effect. 

2.5.1  Weapon  Performance 

Two  different  flash-bang  hand  grenades  (Type  A  and  Type  B)  were  taken  as  candidate  systems.  Physical 
Weapon  Characteristics  (PWC)  are  provided  in  Table  2-4  and  Table  2-5.  The  data  is  obtained  from 
measurements  in  the  open  field  at  ground  level. 


Table  2-4:  PWC  of  Flash-Bang  Hand  Grenade  Type  A 


Distance  to  source  [m] 

- 

1.25 

2.5 

5.0 

10.0 

Peak  light  intensity  [cd/sr] 

8-105 

Peak  sound  level  [kPa] 

35 

12 

5 

3 

Sound  intensity  level  [dB(A)SEL] 

143 

136 

129 

122 
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Table  2-5:  PWC  of  Flash-Bang  Hand  Grenade  Type  B 


Distance  to  source  [m] 


Peak  light  intensity  [cd/sr] 


Peak  sound  level  [kPa] 

Sound  intensity  level  [dB(A)SEL] 


1.25 

5 
128 


2.5 


5.0 


10.0 


125 


119 


115 


The  delivery  characteristics  of  flash-bang  hand  grenades  are  essentially  no  different  from  other  types  of 
hand  grenades.  The  following  values  can  be  used  for  both  types  of  flash-bangs  to  characterise  the  throwing 
performance: 

•  Radius  of  50%  deliveries  from  aimpoint,  distance  15  m,  horizontal  target  area:  1  m. 

•  Radius  of  50%  deliveries  from  aimpoint,  distance  25  m,  horizontal  target  area:  2  m. 

•  Maximum  throwing  distance:  30+5  m. 

Scenario  conditions  are  such  that  the  difference  between  PWC  and  MoP  are  negligible. 

2.5.2  Target  Response 

Permanent  ocular  damage  and  temporary  blindness  are  deemed  to  be  the  most  important  physiological 
responses  resulting  from  bright  flashes  of  light.  For  ‘normal’,  visible  radiation  and  an  exposure  of  0.1  s  it 
is  possible  to  define  threshold  values  for  ocular  damage  and  temporary  flash  blindness.  The  results  are 
summarised  in  Table  2-6. 

Table  2-6:  Target  Response  Thresholds  for  Flashes  of  Light 


Target  Response 

Value  [cd/m2] 

Ocular  damage  (night-time,  foveal  perception) 

8.5-107 

Ocular  damage  (daytime,  foveal  perception) 

2.5-108 

Flash  blindness  for  5  s  (night-time,  foveal  perception) 

3.1-104 

Flash  blindness  for  5  s  (daytime,  foveal  perception) 

4.7- 107 

Generally  applied  threshold  values  for  impulse  sounds  to  prevent  hearing  loss  are  140  dB  sound  pressure 
level  and  125  dB(A)  sound  energy  level.  Regarding  the  required  effective  intensity  of  a  single  impulse 
sound  there  are  three  basic  types  of  responses,  namely  startle,  escape  and  freeze  behaviour.  It  is  possible  to 
produce  relationships  expressing  the  probability  of  each  of  these  responses  as  function  of  the  intensity  of 
the  stimulus.  It  is  also  possible  to  produce  relationships  expressing  the  duration  of  the  responses  as 
function  of  the  intensity  of  the  stimulus.  These  are  the  target  response  relationships,  yielding  MoR. 

2.5.3  Weapon  Effectiveness 

Suppose  the  RR  is  stated  as  follows: 

•  The  primary  military  intent  for  employing  the  flash-bang  device  is  to  degrade  the  target’s  ability 
to  produce  aimed  fire. 
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•  100%  probability  for  a  5  s  degraded  hostile  performance. 

•  2%  probability  of  serious  injury  (i.e.,  loss  of  eyesight  and/or  hearing  loss)  in  Phase  2. 

•  20%  probability  of  serious  injury  (i.e.,  loss  of  eyesight  and/or  hearing  loss)  in  Phase  3. 

The  Basis  Response  that  is  affected  by  the  flash-bang  device,  as  far  as  the  military  intent  is  concerned, 
is  Sense  and  Interpret.  Other  Basis  Responses  that  can  be  directly  affected  are  Motivation  and  Group 
Cohesion.  However,  the  corresponding  target  response  relationships  are  not  available. 

With  the  data  provided  thus  far,  it  is  possible  to  perform  effectiveness  simulations.  The  assessment  of  Pi 
and  P2,  according  to  the  definition  of  SAS-035,  is  provided  in  Table  2-7.  Regarding  P3,  one  may  argue 
that,  as  a  worse  case,  a  non-combatant  can  be  just  as  close  to  a  detonating  flash-bang  hand  grenade  as  the 
intended  target.  Hence,  P3  equals  P2. 


Table  2-7:  MOE  Assessment  of  Flash-Bang  Hand  Grenades 


Flash-Bang 

Pi 

P2 

P3 

Type  A  in  Phase  2 

0.95 

0.62 

0.62 

Type  A  in  Phase  3 

0.95 

0.62 

0.62 

Type  B  in  Phase  2 

0.90 

0.99 

0.99 

Type  B  in  Phase  3 

0.90 

0.99 

0.99 

2.5.4  Issues 

From  the  P2  assessment  one  can  conclude  that  the  flash-bang  Type  B  is  almost  compliant  with  the  military 
requirements  (namely  99%).  However,  one  can  see  that  Type  B  is  in  fact  fully  compliant.  P2  over  time  is  a 
constant  within  the  relatively  short  engagement  timeframe  considered,  and  the  probability  of  hearing 
damage  is  about  1  percent  (1  -  0.99).  Furthermore,  the  definition  of  P2  does  not  result  in  a  different  value 
for  both  types  of  hand  grenades.  This  constitutes  a  problem  since  it  can  be  shown  that  Type  A  can 
intuitively  be  rated  as  ‘becoming  more  compliant  in  Phase  3  as  opposed  to  Phase  2’.  These  observations 
call  for  a  review  of  the  P2  definition. 


2.6  RESULTS  FROM  OTHER  NLW  ASSESSMENTS 

The  results  from  applying  the  methodology  for  the  optical  disrupter,  HPM  weapon,  and  kinetic  energy 
weapons  are  included  as  Annex  C. 

2.7  IMPLICATIONS  AND  ISSUES 

As  noted  previously,  the  Task  1  working  group  identified  several  issues  associated  with  the  P2  and  P3 
calculations.  Figure  2-9  depicts  the  proposed  revision  to  P2  for  the  intended  target  (or,  using  the  typical 
designator,  Red).  Instead  of  calculating  the  shaded  area  above  the  Measured  Response  curve  A  within  the 
rectangle  bounded  by  the  points  cdpq,  P2  is  now  calculated  1  minus  the  area  beneath  curve  A  bounded  by 
the  new  rectangle  cdpq,  where  c  and  q  no  longer  are  at  the  same  magnitude  as  the  required  response  but 
rather  at  the  level  of  the  measured  target  response  at  time  Ti. 
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Old  P2 


New  P2  (RED) 


This  point  is  not  meaningful... 


...and  this  point  is! 


Figure  2-9:  Old  Approach  for  Calculating  P2  versus  New  Proposed  Approach. 

This  change  was  deemed  necessary  because  Pi  and  P2  are  viewed  as  two  distinctly  different  characteristics 
describing  NLW  effectiveness.  The  choice  to  define  the  shared  area  from  T  =  Ti  onwards  by  a  magnitude 
related  to  P!  is  simply  not  correct.  While  the  new  P2  definition  is  viewed  as  an  important  improvement, 
there  is  also  an  issue  with  the  new  P2  that  needs  to  be  addressed.  In  the  new  approach  to  P2  the  upper 
bound  of  the  shared  area  is  defined  by  the  magnitude  of  the  actual  response.  Since  the  actual  response  is  in 
fact  variable,  this  may  complicate  the  process  of  comparing  the  performance  of  one  NLW’s  P2  to  another. 

Moreover,  the  working  group  proposed  eliminating  the  P3  collateral  constraint,  and  replacing  it  with  two 
calculations  -  one  for  own  forces  (Blue)  and  one  for  non-combatants  (White)  -  as  depicted  in  Figure  2-10. 
This  measure  would  calculate  the  shaded  area  under  the  curve  from  the  onset  of  the  (undesired)  response 
until  an  operationally  relevant  recovery  time,  Te. 


New  P2  (BLUE,  WHITE) 


Figure  2-10:  Replacement  of  P3  with  P2  (Blue)  and  P2  (White). 

A  number  of  issues  about  the  use  of  the  methodology  either  arose  in  the  course  of  the  verification  exercise 
or  had  been  previously  identified  and  were  not  fully  resolved  by  the  exercise.  These  include  the  following: 
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1)  Lack  of  Orthogonality  of  the  Basis  Response  Variables.  By  this  is  meant  the  fact  that  there  is  a 
risk  of  overlap  between  different  variables  in  terms  of  what  they  are  measuring,  leading  to  double¬ 
counting  some  of  the  benefits  of  a  given  weapon.  This  problem  arose  in  the  cases  of  the  vehicle 
stopper  -  where  the  estimated  effects  on  Mobility  and  Physical  Function  measure  exactly  the  same 
thing  -  and  the  Optical  Disrupter.  It  is  recommended  that  this  problem  should  be  met  through  a 
combination  of  care  in  the  formulation  of  Basis  Response  variables  and  accounting  for  any 
remaining  non-orthogonality  in  the  process  of  aggregating  across  BRs. 

2)  Choice  of  Appropriate  Basis  Response.  A  related  issue  is  the  difficulty  in  some  cases  of  selecting 
the  right  Basis  Response  variable  to  measure  the  effect  of  a  weapon.  This  arose  in  the  case  of  the 
High  Power  Microwave  weapon,  where  the  effect  could  equally  well  be  defined  in  terms  of  the 
Communication  or  Sense  and  Interpret  responses,  and  arguably  in  terms  of  Physical  Function  as 
well.  A  more  orthogonal  and  explicit  definition  of  the  Basis  Responses  is  a  partial  solution. 

3)  Provision  of  Required  Response.  The  RR  specifies  the  Commander’s  requirement  for  the 
weapon,  and  should  ideally  come  from  military  sources.  All  the  RR  used  in  this  exercise  came  in 
fact  from  scientists,  and  the  exercise  has  not  fully  verified  that  it  is  possible  to  cany  out  the 
methodology  using  RR  from  military  sources.  It  was  agreed  in  the  course  of  the  study  that  the  RR 
needs  to  be  specified  in  terms  of  physically  meaningful  and  measurable  variables  conforming  to 
the  description  of  the  methodology  in  the  SAS-035  report,  in  terms  of  time  profiles  of  effect  on  the 
Basis  Responses.  It  was  felt  that,  typically,  commanders  will  state  their  desired  effect  on  target 
behaviour,  and  the  methodology  should  be  able  to  translate  this  statement  into  a  combination  of 
Basis  Response  levels.  Note  that  more  than  one  combination  of  levels  may  achieve  the  desired 
effect.  The  demonstration  of  such  a  process  has  not  been  achieved  in  SAS-060. 

4)  Coupled  Effects.  The  possibility  was  identified  that,  where  more  than  one  weapon  was  used  in 
achieving  the  same  task  the  effects  might  interact,  with  the  result  that  the  overall  effect  would  be 
more  than  the  sum  of  the  individual  effects.  For  example,  in  the  use  of  a  paintball  and  vehicle 
stopper  sequentially  to  stop  a  car,  a  partly  obscured  front  window  leaves  the  driver  with  less 
opportunity  to  counter-act  the  next  obstacle.  The  initial  performance  of  the  car-plus-driver  is  partly 
disrupted,  and  this  reduced  level  of  performance  is  the  starting  point  for  the  next  engagement 
(e.g.,  vehicle  stopper  deployment). 

This  issue  can  be  assigned  to  the  MoOE  part  of  the  methodology.  However,  it  is  equally  possible 
to  think  of  two  weapon  effects  acting  simultaneously,  during  a  1-on-l  engagement,  making  it  a 
MoSE  issue. 

5)  Derivation  of  MoR  from  MoP.  MoP  were  available  for  all  the  weapons  examined  in  this 
exercise,  but  it  is  noticeable  that  the  only  ones  for  which  complete  MoR  were  available  were  those 
-  vehicle  stopper  and  HPM  -  whose  effect  is  on  equipment  rather  than  people.  Deriving  MoRs  for 
human  subjects  requires  either  a  validated  theory  -  which  is  not  known  to  exist  in  most  cases  - 
or  experiments  -  which  raise  ethical  and  practical  problems.  One  possible  exception  is  flash- 
bangs,  where  much  experimental  data  is  available.  Because  of  ethical  and  experimental  problems, 
in  some  cases,  e.g.,  kinetic  energy  weapons,  field  data  (post-use  audits)  may  be  the  only  means  of 
assessing  effectiveness.  In  general  the  exercise  cannot  be  said  to  have  shown  that  MoR  data  will 
usually  be  available. 

6)  Issues  Related  to  MoSE.  These  issues  include: 

•  MoSE  normalisation  -  in  order  to  calculate  the  MoSE  measures  and  normalise  them  to  lie  in 
[-1,1]  the  methodology  requires  specification  of  a  total  time  period  for  the  scenario,  and  upper 
limits  to  the  effects  achieved  by  the  weapon.  If  the  weapon  effect  does  not  return  to  zero  in 
some  reasonable  time  then  (as  demonstrated  by  the  case  of  the  vehicle  stopper)  the  values  of 
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P2  and  P3  will  depend  on  the  time  period  selected  as  the  length  of  the  scenario,  and  an  agreed 
method  of  defining  the  scenario  cut  off  point  is  needed. 

Setting  the  upper  limit  to  the  effect  did  not  in  practice  cause  problems  in  the  exercise,  but  there 
are  at  least  three  issues  here: 

•  In  the  case  of  Pi,  how  to  account  for  cases  where  the  weapon  has  a  greater  effect  than 
the  specified  upper  limit. 

•  In  the  case  of  P2,  where  the  aim  is  to  eliminate  the  effect  as  quickly  as  possible  when 
it  is  no  longer  needed,  a  strong  argument  can  be  made  that  excess  effects  should  be 
included;  but  this  could  lead  to  a  value  of  P2  <  - 1 . 

•  In  the  case  of  P3,  there  is  no  obvious  way  of  setting  the  upper  threshold,  and  so  it  is 
not  obvious  how  to  set  an  upper  limit  to,  and  hence  normalise,  collateral  effects. 

•  Whether  the  methodology  should  continue  to  incorporate  P3.  In  the  course  of  the  study 
concerns  arose  that  P3  (collateral  MoSE)  was  almost  immeasurable  in  its  current  form.  It  was 
suggested  that  a  better  approach  would  be  to  repeat  an  analysis  of  Pj  and  P2  for  Red,  Blue  and 
White  actors.  In  this  way  a  series  of  acceptable  effect  magnitudes,  durations  and  recovery 
times  could  be  specified  for  all  actors.  This  removes  the  arbitrariness  of  the  “upper  magnitude 
comparison  limit”  and  makes  all  calculations  more  logical  and  defensible. 

These  issues,  about  normalisation  and  the  definition  of  P3,  resulted  in  recommendations  for 
the  redefinition  of  P2  and  P3,  as  previously  discussed. 

7)  Sensitivity  Analyses.  Whilst  the  vehicle  stopper  exercise  has  demonstrated  the  feasibility  of 
sensitivity  analyses  within  the  methodology,  the  more  general  use  of  sensitivity  analyses  remained 
an  issue  at  the  end  of  Task  1,  as  it  depends  on  estimates  being  available  of  the  relative  uncertainty 
in  the  various  data  -  RR,  MoR  and  the  thresholds  used  in  calculating  Pl5  P2  and  P3  -  and  the 
feasibility  of  getting  such  estimates  has  not  been  demonstrated. 

8)  Software  Implementation.  The  SAS-035  report  envisioned  the  implementation  of  the 
methodology  as  a  computer  model.  However,  the  experience  of  Task  1,  applying  the  methodology 
to  a  diverse  range  of  weapons,  suggests  that  something  simpler  and  less  formalised  is  needed. 
The  methodology  is  best  viewed  as  a  thought  process  for  carrying  out  one  stage  in  the  process  of 
assessing  and  comparing  non-lethal  weapons,  to  be  adapted  as  appropriate  to  the  particular  set  of 
weapons  being  studied.  No  single  computer  tool  would  meet  the  needs  of  all  weapon  types.  Given 
the  ability  to  specify  the  MoP,  MoR  and  RR  in  agreed  formats,  then  a  calculation  engine  could  be 
created  which  would  perform  the  various  MoSE  calculations.  However,  the  uncertainty  over  the 
data  sets  and  fields  makes  the  creation  of  such  a  tool  difficult. 

9)  Use  of  Combat  Modelling.  The  possibility  was  identified  that  aggregation  methods  of  deriving 
MoSE  from  MoOE  would  not  adequately  capture  the  impact  of  effects  to  individual  targets  on  the 
course  of  the  scenario  as  a  whole  -  that,  for  instance  a  small  effect  on  one  key  enemy  activity  may 
have  a  major  effect  on  later  events.  In  that  case,  the  MoSE  measures  could  only  be  integrated 
adequately  into  a  meaningful  high  level  measure  of  operational  success  by  formal  modelling  or 
wargaming  of  the  overall  sequence  of  events  in  a  scenario  in  which  the  weapon(s)  are  used. 


2.8  TASK  1  CONCLUSIONS  AND  RECOMMENDATIONS 

The  following  conclusions  emerged  from  Task  1: 

1)  The  exercise  has  shown  that  the  methodology  is  verifiable  by  the  process  adopted  -  that  is,  that  there 
is  no  reason  to  believe  the  process  could  not  be  verified  if  it  were  completely  carried  through. 
In  fact: 
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•  A  number  of  issues  with  the  methodology  have  been  identified,  but  at  present  none  are 
believed  to  be  “show-stoppers”. 

•  The  feasibility  of  carrying  out  sensitivity  analysis  has  been  demonstrated  for  one  weapon 
system. 

2)  The  methodology  should  currently  be  regarded  as  a  thought  process,  which  should  be  adapted  to 
the  needs  of  assessing  specific  weapon  systems,  rather  than  a  formal  model  which  should  be 
implemented  in  software. 

3)  Issues  remain  with: 

•  The  lack  of  orthogonality  of  the  Basis  Response  variables. 

•  Choice  of  appropriate  Basis  Response  for  some  weapon  types. 

•  Provision  of  Required  Response  from  military  sources. 

•  Derivation  of  Measures  of  Response  from  Measures  of  Performance. 

•  Issues  related  to  Measures  of  System  Effectiveness  -  including  normalisation,  definition  of 
scenario  endpoint  and  how  to  measure  collateral  effects. 

•  Sensitivity  analyses. 

•  Software  Implementation. 

•  Coupled  effects. 

•  The  possible  use  of  combat  modelling  as  an  aggregation  method. 

It  is  recommended  that  the  SAS-035  framework  should  be  modified,  that  a  modified  version  of  the  P2 
measure  of  undesired  effects  should  be  applied  universally  to  Blue,  Red  and  White  participants,  and  that 
the  P3  measure  should  be  removed  from  the  methodology. 
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Chapter  3  -  AGGREGATING  RESULTS  FOR  SYSTEM 
AND  OPERATIONAL  EFFECTIVENESS 

3.1  INTRODUCTION 

The  Task  2  working  group  had  two  linked  objectives,  extending  the  Measures  of  System  Effectiveness 
methodology  and  exploring  development  of  Measures  of  Operational  Effectiveness.  Supporting  objectives 
included: 

•  Addressing  issues  carrying  over  from  Task  1  regarding  the  MoSE  methodology  (as  refined). 

•  Analysing  additional  scenarios. 

•  Identifying  distinctions  between  system  and  operational  level  measures  of  effectiveness. 

•  Determining  potential  methods  of  aggregating  results  -  at  a  single  point  in  time  and  across  phases 
of  a  scenario  -  and  comparing  the  methods. 

•  Analysing  potential  sources  of  variation  and  confidence  levels. 

3.2  INPUTS  FROM  TASK  1 

The  starting  point  for  Task  2  efforts  were  the  outputs  from  the  Task  1  working  group: 

•  The  refined  MoSE  methodology  as  documented  in  a  Task  1  report. 

•  System  effectiveness  data  generated  as  part  of  the  Task  1  assessments: 

•  Vehicle  Arrestor, 

•  Optical  Disrupter, 

•  High-Power  Microwave  Weapon, 

•  Kinetic  Energy  Weapons, 

•  Flash-Bangs. 

•  Key  issues  identified  (see  Section  2.7). 


3.3  SCENARIO  ANALYSES 

The  Task  2  working  group  analysed  additional  scenarios: 

■  Urban  Operations  2020  Warfighting  Scenario  -  This  was  a  fairly  high  intensity  scenario  with 
factions  fighting  in  the  aftermath  of  a  government  collapse  and  a  mission  to  stabilise  the  country  and 
restore  territorial  integrity,  reduce  potential  for  regional  conflict,  and  ameliorate  a  humanitarian 
crisis.  NLW-related  tasks  were  associated  with  limiting  collateral  damage  and  reconstmction 
requirements  as  well  as  security  operations  in  conjunction  with  humanitarian  efforts. 

■  Urban  Operations  2020  Contingency  Response  Operation  Scenario  -  This  had  many  similar 
elements  to  the  previous  scenario  but  with  a  cease  fire  agreement  and  Chapter  VII  UN  Security 
Council  Resolution  in  place.  The  NLW-relevant  tasks  were  the  same  as  in  the  warfighting  scenario. 

■  Peace  Enforcement  -  This  scenario  involved  NATO  Peace  Enforcement  in  a  non-permissive 
environment,  with  relevant  tasks  including  NLW  use  as  part  of  offensive  operations,  crowd 
dispersal,  area  denial  to  personnel,  and  crowd  and  riot  control  (with  specific  tasks  to  incapacitate, 
control  movement/repel/employ  barriers,  and  mark/identify). 
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■  Deny  Use  of  WMD  -  In  a  two-sided  civil  war  having  the  threat  of  chemical  weapons  use, 
the  mission  is  to  prevent  WMD  use,  ameliorate  the  humanitarian  crisis,  and  minimise  loss  of  life 
and  chance  of  CW  release.  Given  the  risk  of  a  chemical  release,  NLW  may  apply  in  many  tasks, 
including:  prevent  the  manufacture  of  chemical  agents,  render  CW  facilities  inoperable,  locate 
and  distinguish  combatants  and  non-combatants,  neutralise  hostile  and  potentially  hostile  troops, 
clear  CW  facility  of  personnel,  deny  enemy  access  to  CW  facility  and  deny  areas,  disrupt 
communications,  disperse  non-combatants  and  clear  areas,  and  provide  personnel  and  facility 
security. 

■  Urban  Operations  (Military  Disaster  Relief)  -  Following  a  major  earthquake,  the  mission  is  to 
conduct  non-combatant  evacuation  opeations,  distribute  emergency  supplies,  and  evacuate/ 
relocate  refugees  and  displaced  persons.  Specified  NLW-relevant  tasks  are  crowd  and  riot  control, 
distinguish  between  hostile  and  non-hostile,  erect  barriers,  control  movement,  and  deny  areas. 

■  Urban  Operations  ( Peacekeeping )  -  In  the  aftermath  of  extended  conflict  over  autonomy, 
the  mission  is  to  maintain  order  and  a  secure  environment,  confiscate  weapons,  provide  security 
for  NGOs,  and  safeguard  citizens  while  minimising  civilian  casualties  and  collateral  damage. 

■  Urban  Operations  (Urban  Combat)  -  This  was  a  high  intensity  scenario  involving  a  powerful 
state  pursuing  regional  hegemony  including  through  subversion  of  its  neighbors.  This  scenario 
indicates  an  important  role  for  NLW  as  the  mission  explicitly  cites  the  need  to  minimise  casualties 
and  collateral  damage  while  regaining  control  of  the  government  capital. 

■  Humanitarian  Assistance  -  Following  governmental  collapse  a  third  world  country  faces 
spreading  famine  and  chaos.  The  mission  is  to  maintain  general  and  local  area  security,  distribute 
essential  supplies,  and  facilitate  logistics  with  the  following  specified  NLW-relevant  tasks:  open 
and  maintain  ingress/egress  routes;  control  movement,  ability  to  act/shoot,  etc.;  deny  areas/deny 
access  to  individuals  and  vehicles;  incapacitate  individuals;  stop  and/or  neutralise  vehicles;  mark 
targets;  restrict  communications  (leader  to  crowd);  and  minimise  civilian  casualties  and  collateral 
damage. 

For  each  scenario,  the  Task  2  working  group  analysed  the  mission,  threats,  environmental  factors, 
specified  tasks,  task-target  combinations,  linkages  (Primary  and  Secondary)  to  Basis  Responses  (depicted 
in  Table  3-1),  desired  response  characteristics,  and  engagement  characteristics. 
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Table  3-1 :  Linkages  from  Tasks  to  Basis  Responses  (BRs) 


Deny  Area/Deny  Access  to  Individuals 


Secure  and  Defend  Against  Individuals 


Contain,  Isolate,  Separate  Individuals 


Delay  Individuals 


Incapacitate  Individuals 


Disperse  Individuals 


Distract  and/or  Disorient  Individuals 


Degrade/Suppress  Individuals 


Canalize  Individuals 


Mobility  Communications  Physical  Group  Sense  &  Motivation  Identification 

Function  Cohesion  Interpret 


Clear  Buildings/Facilities/Structures  of  Individuals 


Clear  Vessels/Aircraft  of  Personnel 


Interrogate  Intent  of  Individuals 


Deceive/Demonstrate  to  Individuals 


S 
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Disable  Vehicles,  Vessels,  and  Aircraft 

P 

Render  Inoperable/Deny  Use  of  Vehicles,  Vessels, 
and  Aircraft 

P 

Stop/Interdict  Vehicles/V essels/ Aircraft 

P 

Render  Inoperable/Deny  Use  of  Weapons 

Degrade/Suppress  Weapon  Systems  and  Ammunition 

Render  Inoperable/Deny  Use  of  Vehicles,  Vessels, 
and  Aircraft 

p 

Contain/Isolate/Separate/Delay  Movement  from 
Buildings,  Facilities,  and  Structures 

P 

Deny  Area/Deny  Access  to  Vehicles,  Vessels,  and 
Aircraft 

P 

Render  Inoperable/Deny  Use  of  Buildings,  Facilities, 
and  Structures 

Render  Inoperable/Deny  Use  of  Buildings,  Facilities, 
and  Structures 

Contain/Isolate/Separate  Vehicles,  Vessels,  and 
Aircraft 

P 

Delay  Vehicles,  Vessels,  and  Aircraft 

P 

Canalize  Vehicles  and  Vessels 

p 

Deny  Area/Deny  Access/Render  not  Trafficable  to 
Vehicles,  Vessels,  &  Aircraft 

p 
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Render  Inoperable/Deny  Use  of  WMD 


Render  Inoperable/Deny  Use  of  C4ISR 


Render  Inoperable/Deny  Use  of 
Buildings/Facilities/Structures 


Degrade/Suppress  C4ISR 


3.4  MoSE  AND  MoOE  DEFINITIONS/DISTINCTIONS 

The  Task  2  working  group  sought  to  identify  distinctions  between  MoSEs  and  MoOEs  by  examining 
differences  between  tactical  and  operational  level  tasks,  with  the  underlying  intent  to  facilitate  development 
of  MoOEs.  However,  there  were  no  major  differences  in  the  types  of  tasks  as  illustrated  in  Table  3-2. 

Table  3-2:  Operational  and  Tactical  Level  Task  Comparisons 

Task  Categories 

Operational  Level  Conduct  Operational  Movement/Manoeuvre 

Employ  Operational  Firepower 
Provide  Operational  Logistics/Personnel  Support 
Provide  Operational  Force  Protection 


Tactical  Level 

Deploy /Conduct  Manoeuvre 

Employ  Firepower 

Provide  Combat  Service  Support 

Protect  the  Force 
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The  differences  between  operational  and  tactical  level  tasks  are  not  in  the  nature  of  the  tasks  but  rather 
their  scope  and  scale.  In  defining  required  outcomes  (and  MoOEs)  as  opposed  to  required  responses  (and 
MoSEs),  Table  3-3  shows  useful  distinctions. 


Table  3-3:  MoSE  vs.  MoOE  Distinctions 


Required  Response/MoSE 

Required  Outcome/MoOE 

Single  Task 

Multiple  Tasks 

Single  Target 

Multiple  Targets 

Single  NLW 

Multiple  NLW 

Single  Phase 

Extended  Duration  (across  phases) 

3.5  AGGREGATION  APPROACHES  AND  EXAMPLES 
3.5.1  Methods  and  Description 

The  methods  proposed,  studied,  and  implemented  for  aggregating  effectiveness  can  be  classified  into  the 
following  categories: 


ANALYTICAL 

"  SAS-035 

FORMULATION 

fGeneralization:  Weighted  Minkowski  Distance 
Weighted  Euclidean  Distance  -< 

MULTI- 

r  IParticularization:  Root  Mean  Square 

Scoring  Model 

CRITERIA  J 

Pairwise  Comparison  Weighted  ^Variant:  Random  Weights  Generation  averaged 

ARTIFICIAL 

Analytic  Hierarchy  Process 

r~ 

NEURAL  J 

Supervised  Radial  Basis  Function  Network 

NETWORK 

All  the  methods  are  normalized  in  order  to  obtain  aggregated  values  between  0  and  1 ,  allowing  for  direct 
comparisons  among  methods. 


3.5.1.1  Analytical  Formulations 


3. 5. 1.1.1  SAS-035  Proposed  Formula 

SAS-0351  anticipated  the  need  to  aggregate  effectiveness  results  (whether  for  task  accomplishment,  Pi, 
or  constraint  satisfaction,  P2)  across  basis  responses.  An  example  analytical  formulation  was  expressed  as 
follows: 


ID 


r  Pi +  \P^ 


^  +  < 


1  + 


V 


1  SAS-035  final  report,  pp.  3-19. 
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This  formula  is  easy  to  use.  However,  the  formulation  has  an  important  limitation:  saturation.  If  any  P;  =  1 
(and  all  are  non-negative),  then  the  aggregated  P*  =  1,  making  the  other  basis  response  (BR)  values 
irrelevant.  Nevertheless,  this  method  is  included  for  comparison  purposes. 

3. 5. 1.1.2  Weighted  Euclidean  Distance  (WED) 

The  Weighted  Euclidean  Distance  in  a  seven-dimensional  space  (number  of  BRs)  is  a  simple  and 
straightforward  method  to  solve  the  aggregation  problem.  This  method  takes  into  account  BRs’  relative 
importance  through  a  vector  of  weights  w;  (i  =  1,2,  ...  ,7).  Weighting  is  determined  by  the  user  and  the  sum 
of  all  the  weights  must  add  up  to  one. 

The  mathematical  formulation  of  this  aggregation  method  is  as  follows: 

7 

^  wi  =  1 ;  With  Wj  >  0 

i=i 


where: 


d(O.P) 

d(0,E) 


Wj'.  vector  of  normalized  weights; 


O  (0,0,0, 0,0, 0,0):  origin  with  all  effectiveness  values  of  the  BRs  equal  to  zero;  and 


E  (1,1, 1,1, 1,1,1):  maximum  value  (all  effectiveness  values  of  the  BRs  equal  one). 


When  the  relative  importance  is  the  same  for  all  the  BRs,  then  the  aggregation  equation  (called  Averaged 
Euclidean  Distance)  is  equal  to  the  Root  Mean  Square  (RMS).  The  RMS,  utilized  in  many  research 
areas,  is  a  measure  of  the  magnitude  of  a  set  of  numbers  that  gives  a  sense  of  the  typical  size  of  the 
numbers.  The  RMS  aggregated  value  (P*)  for  the  effectiveness  of  the  seven  Basis  Responses  {Pi,  P2,  ..., 
P  !  is: 


The  RMS  is  an  appropriate  aggregation  method  when  the  user  is  not  interested  in  the  relative  priority  of 
different  Basis  Responses.  When  the  user  wants  to  rank  the  importance  of  the  BRs,  RMS  is  not  a  suitable 
method. 


The  WED  is  a  particularization  of  the  Weighted  Minkowski  Distance.  The  WMD  of  order  k  can  be  used  as 
an  aggregation  method: 


P* 


\i= 1 


1 

k 


J 


When  the  Minkowski  factor  for  the  norm  k  (order)  is  set  as  2,  the  expression  is  the  WED.  When  k  is  set  to 
1,  the  formula  is  a  linear  weighting  model  (Scoring  Model). 
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3.5.1.2  Multi- Criteria  Decision  Models  (MCDM) 

Multi-Criteria  Decision  Models  (MCDM)  are  methods  for  solving  decision  problems.  In  this  section, 
MCDM  are  used  to  aggregate  effectiveness  and  rank  alternatives  (NLW).  Hence,  BRs  are  weighted  by 
coefficients  that  measure  relative  importance  based  on  decision  makers’  experience. 

3. 5. 1.2.1  Scoring  Model  (SM) 

The  Scoring  Model  (SM)  is  a  simple  and  fast  method  to  solve  the  aggregation  problem.  The  critical  step  is 
determining  appropriate  weights  for  the  BRs  for  different  situations  (scenario,  phase  of  engagement, 
requirements,  etc.).  The  relative  weights  must  be  assigned  based  on  the  concrete  objectives  of  each 
operation,  with  the  sum  of  all  the  weights  must  be  equal  to  one  (aggregated  values  are  bounded  to  one). 

The  steps  of  the  SM  are  the  following: 

1)  Identify  the  main  goal  Choose  the  best  NLW. 

2)  Identify  the  alternatives  Different  NLW. 

3)  List  the  criteria  to  apply  in  making  the  decision  Different  BRs. 

4)  Assign  the  weights  for  the  different  criteria  Weights  Vector. 

5)  Calculate  the  Score  for  each  alternative. 

6)  Rank  the  alternatives  according  to  the  Score.  The  NLW  with  the  highest  Score  is  the  best. 

The  scoring  formula  for  j  types  of  NLW  is: 

r;=Tw<-p« 

1=1 


Where: 


Pjj  =  Effectiveness  value  of  the  BRi  of  the  NLWj 
Wj  =  weight  for  the  BRi 
P  j  =  Aggregated  value  for  the  NLW , 

Therefore,  the  SM  is  a  basic  method  to  aggregate  effectiveness  when  the  relative  importance  of  Basis 
Responses  matters.  When  the  relative  importance  is  the  same  for  all  BRs,  then  the  outcome  is  an  average 
(arithmetic  mean). 


3.5. 1.2.2  Pairwise  Comparisons  Weighted  (PCW) 

The  PCW  method  compares  pairs  of  input  values  (referred  to  as  pairwise  comparisons),  subtracting  one 
from  all  the  others.  Then  weights  must  be  assigned,  with  knowledge  from  experts  used  to  build  the 
Weights  Vector  (WV). 


The  pairwise  dominance  might  be  checked  with  the  following  LP  models  used  to  make  the  aggregation: 


P  =  max 


i>,(L 


V  1=1 


Where:  P;I  is  the  effectiveness  value  of  the  NLW,  with  respect  to  BR;  and  the  maximization  is  over  the 
subindex  k. 


3-8 


RTO-TR-SAS-060 


AGGREGATING  RESULTS  FOR 
SYSTEM  AND  OPERATIONAL  EFFECTIVENESS 


The  weights  take  values  between  0  and  1  and  they  must  add  up  to  1.  So,  the  aggregated  results  are 
bounded  by  1.  Every  specific  case  has  a  different  Weights  Vector  related  to  it,  and  the  experience  of 
working  with  the  WV  is  very  important. 

A  method  based  on  the  PCW  formulation,  is  the  Random- Weights  Generation  averaged  (RWG). 
This  method  does  not  apply  user  criteria.  Instead,  weights  are  generated  using  random  uniform 
distributions  for  the  simulation  of  additive  i-criteria  decision  models  (i  is  the  number  of  BRs).  In  the 
procedure,  i-1  random  numbers  are  first  selected  from  a  uniform  distribution  on  (0,  1)  independently. 
Then,  the  numbers  are  ranked.  Suppose  the  ranked  numbers  are  1  >  v,-./  >  ...  >  v2  >  Vj  >  0.  The  first 
differences  of  these  ranked  numbers  (including  the  bounds  0  and  1)  can  be  obtained  as: 

Wi=  1  -  Vi-i 
Wj-I  =  Vi-1  -  Vi-2 


vv/  =  V]  -  0 

In  this  case,  the  set  of  numbers  (wj,  W2,  . ..,  w~)  will  clearly  add  up  to  one.  Once  the  weights  have  been 
generated,  the  aggregation  PCW  formula  is  applied. 

Moreover,  the  aggregated  value  for  each  j  must  be  calculated  T  times  (Monte  Carlo  simulations)  to  obtain 
an  average  outcome: 


If  the  value  of  T  is  high  enough,  then  the  relative  importance  given  to  all  BRs  tends  to  be  the  same. 
Therefore,  the  results  of  both  methods  are  the  same  when  RWG  is  calculated  with  a  high  T  and  the 
weights  vector  gives  the  same  importance  to  all  BRs  in  the  PCW  case. 

When  knowledge  from  experts  is  not  available,  the  RWG  can  be  used  to  generate  stochastic  aggregation 
results  for  comparison  purposes. 

3. 5. 1.2.3  Analytic  Hierarchy  Process  (AHP) 

The  Analytic  Hierarchy  Process  (AHP)  can  be  used  in  single  or  multi-dimensional  decision-making 
problems.  It  is  a  methodology  suitable  for  aggregation  and  for  determining  weights.  AHP  is  based  on 
decomposing  a  complex  MCDM  problem  into  a  system  of  hierarchies  (with  an  example  presented  in 
Annex  D). 

Owing  to  its  simplicity  and  ease  of  use,  the  AHP  has  found  ready  acceptance  by  decision-makers.  It  helps 
structure  thoughts  and  organise  problems  in  a  manner  that  is  simple  to  follow  and  analyse.  Basically  the 
AHP  helps  in  structuring  the  complexity,  measurement  and  synthesis  of  rankings,  features  that  make  it 
suitable  for  our  purposes. 

The  matrices  arc  (7xN),  seven  being  the  number  of  criteria  (BRs)  and  N  the  number  of  alternatives  (NLW), 
for  a  specific  Pk. 

This  matrix  is  constructed  by  using  the  relative  importance  of  the  NLW  in  terms  of  each  BR.  The  vector 
(aij,  ay,  a3j,  ...,  ay)  for  each  j  is  the  principal  eigenvector  of  a  7x7  reciprocal  matrix  which  is  determined 
by  pairwise  comparisons  of  the  impact  of  the  N  NLW  on  the  i-th  BR. 
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Figure  3-1 :  AHP  Diagram. 


The  matrix  (7xN)  is  aggregated  in  a  (Nxl)  vector.  This  vector  represents  the  specific  Pk  aggregated  for  the 
N  different  types  of  NLW. 

The  AHP  produces  weight  values  for  each  alternative  based  on  the  judged  importance  of  one  NLW  versus 
another2  with  respect  to  a  common  basis  response.  Its  main  value  resides  in  the  easy  and  logical  method  of 
obtaining  the  weights. 

3.5.1.3  Artificial  Neural  Networks  (ANN) 

Artificial  Neural  Networks  (ANNs)  have  been  used  in  multiple  research  areas  and  can  be  adapted  to 
aggregation  (see  Annex  D  for  a  detailed  description). 

Several  options  were  studied  and  analyzed  to  solve  our  problem  of  aggregation  considering  unsupervised 
and  supervised  nets.  Finally  the  net  proposed  and  implemented  to  solve  the  problem  of  aggregation  was  a 
supervised  radial  basis  function  network. 


2 

Knowledge  from  experts/commanders  is  needed  to  build  the  Criteria  Comparison  Matrix.  Every  specific  case  has  a  Criteria 
Comparison  Matrix  related  to  it,  and  experience  is  very  important. 
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Figure  3-2:  Supervised  Radial  Basis  Net  Block  Diagram. 


The  target  (known  goal  output  state)  to  train  the  net  is  obtained  by  aggregation  methods  which  need  user 
criteria.  The  net  is  supervised  because  the  criteria  are  taken  into  account  when  obtaining  the  corresponding 
target  to  train  the  net.  This  target  is  an  average  among  all  the  results  obtained  by  the  other  “weighted” 
methods  (SM,  WED,  and  AHP).  Thereby,  the  ANN  results  are  “combinations”  of  the  other  weighted 
methods’  aggregated  values. 

Once  there  is  a  trained  net,  the  ANN  method  easily  calculates  results  for  different  scenarios,  engagement 
phases,  etc.  The  SM,  WED,  and  AHP  are  no  longer  required  because  the  net  is  already  trained,  and  results 
can  be  obtained  directly. 

Given  an  input  matrix  (7xN),  the  network  yields  an  output  vector  Nxl  with  the  aggregated  values. 

3.5.2  Comparison  of  Methods 
3.5.2.1  Comparison  of  Results 

The  MoSE  are  split  into  three  different  types:  intended  target  (Red),  our  own  forces  (Blue)  and  non¬ 
combatants/others  (White).  Every  colour  is  treated  in  the  same  way  but  independently  from  each  other. 
The  results  presented  in  the  comparison  are  only  for  P|  for  Red. 

The  following  table  (Table  3-4)  presents  data  on  six  NLW-target  combinations:  Optical  Disrupter,  vehicle 
stopper  (30,  49  and  70  mph),  and  kinetic  weapons  (Blinis  and  12  gauge).  The  six  Pi  matrices  (7x2)  available 
(colour  red)  are: 
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Table  3-4:  NLW  Input  Data 


RED 

NLW  1: 

Optical 

Disrupter 

NLW  2: 

Vehicle 
Stopper 
(30  mph) 

NLW  3: 

Vehicle 
Stopper 
(49  mph) 

NLW  4: 

Vehicle 
Stopper 
(70  mph) 

NLW  5: 

Kinetic 

(Blinis) 

NLW  6: 

Kinetic 
(12  gauge) 

BR 

Pi 

P2 

Pi 

P2 

Pi 

P2 

Pi 

p2 

Pi 

P2 

Pi 

P2 

Mobility 

0.8 

0.6 

0.9959 

0.5 

0.9748 

0.5 

0 

1 

0.75 

0.6 

0.4 

1 

Communication 

0 

1 

0 

0 

0 

0 

0 

0 

0.2 

1 

0.1 

1 

Physical  Function 

0.8 

0.6 

0.9959 

0.5 

0.9748 

0.5 

0 

1 

0.5 

1 

0.8 

1 

Sense/Interpret 

1 

0.75 

0 

0 

0 

0 

0 

0 

0.95 

0.65 

0.4 

1 

Group  Cohesion 

0.75 

0.75 

0 

0 

0 

0 

0 

0 

0.75 

0.8 

0.6 

0.9 

Identification 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Motivation 

0 

0 

0 

0 

0 

0 

0 

0 

0.4 

1 

0.4 

0.9 

In  order  to  compare  the  different  aggregation  methods,  the  effectiveness  values  of  the  BRs  are  equally 
weighted  when  user  weights  are  needed. 

Results  obtained  by  the  different  methods  are  shown  in  order  to  clarify  the  conclusions: 


□  Optical  Disrupter  □  X-Net  (30  mph)  □  X-Net  (49  mph)  □  X-Net  (70  mph)  □  Kinetic  (Blinis)  □  Kinetic  (12  gauge) 


Figure  3-3:  Comparison  of  Results. 
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Table  3-5:  Methods’  Results 


p*t 

SAS-035 

RMS 

WED 

SM 

RWG 
(N  =  25) 

PCW 

AHP 

ANN 

NLW  1 :  Optical  Disrupter 

1.0000 

0.6372 

0.6372 

0.4786 

0.4617 

0.4786 

0.1933 

0.4364 

NLW  2:  Vehicle  Stopper 
(30  mph) 

1.0000 

0.5323 

0.5323 

0.2846 

0.2881 

0.2846 

0.0958 

0.3043 

NLW  3:  Vehicle  Stopper 
(49  mph) 

0.9994 

0.5211 

0.5211 

0.2786 

0.2820 

0.2786 

0.0943 

0.2980 

NLW  4:  Vehicle  Stopper 
(70  mph)3 

0 

0 

0 

0 

0 

0 

0.0247 

0.0082 

NLW  5:  Kinetic  (Blinis) 

0.9992 

0.5949 

0.5949 

0.5072 

0.4669 

0.5072 

0.3424 

0.4815 

NLW  6:  Kinetic  (12  gauge) 

0.9844 

0.4614 

0.4614 

0.3857 

0.3506 

0.3857 

0.2496 

0.3656 

The  S AS-035  formula  results  show  the  saturation  issue.  If  any  P,  =  I  (and  all  Pi  are  non-negative),  then  the 
aggregated  P  =  1 ,  as  seen  with  the  Optical  Disrupter. 

The  RMS  does  not  have  a  saturation  issue.  However,  it  does  not  take  into  account  the  priority  among  BRs, 
and  this  priority  is  important  while  analysing  a  particular  sequence  of  events.  This  method  is  adequate 
when  BR  criteria  are  not  needed.  When  all  the  weights  are  equal,  the  results  obtained  by  the  WED  are  the 
same  as  the  RMS. 

As  with  the  WED,  the  SM  also  takes  into  account  the  priority  of  the  BRs.  As  explained  before,  as  all  the 
weights  are  equal,  the  aggregated  values  are  the  arithmetic  mean  of  the  effectiveness  values  of  the  BRs. 

The  results  of  the  SM  and  the  PCW  are  the  same  because  the  maximum  pairwise  comparison  in  the  PCW 
is  made  with  the  vehicle  stopper  (70  mph),  and  its  inputs  are  0  for  all  the  effectiveness  values  of  the  BRs. 

In  this  case,  as  all  the  weights  are  equal,  the  RWG  gives  aggregated  results  similar  to  the  PCW  (the  higher 
the  number  of  simulations  for  the  RWG,  the  more  similar  the  results  would  be). 

In  the  AHP,  the  sum  of  the  aggregated  values  for  all  the  NLWs  considered  is  equal  to  1.  This  is  why  the 
aggregated  values  are  lower  than  in  the  other  methods  and  depends  on  the  number  and  types  of  NLWs 
selected.  For  this  reason,  an  accurate  comparison  of  AHP  results  with  the  other  methods  could  only  be 
made  by  scaling  other  methods  with  the  number  of  NLW  available. 

Finally,  the  ANN  method  is  a  combination  among  “weighted  methods”  (WED,  SM,  and  AHP)  results. 
Once  the  training  is  completed,  the  aggregated  values  provided  are  the  average  of  these  three  methods. 

3.5.2.2  Task-Target  Combination  Aggregation 

Here  is  an  example  using  Table  3-1  (Linkages  from  Tasks  to  Basis  Responses).  The  aggregation  methods 
will  be  applied  to  the  “incapacitate  individuals”  case  (consequently,  the  NLWs  used  need  to  be  anti¬ 
personnel). 


3 


The  Vehicle  stopper  (70  mph)  result  is  null  because  the  Pj  inputs  are  0  (see  inputs  in  Table  2-2). 


RTO-TR-SAS-060 


3-13 


AGGREGATING  RESULTS  FOR 

SYSTEM  AND  OPERATIONAL  EFFECTIVENESS 


ORGANIZATION 


Table  3-6:  Linkages  from  Tasks  to  BRs:  Incapacitate  Individuals 


Mobility 

Communi¬ 

cations 

Physical 

Function 

Group 

Cohesion 

Sense  & 
Interpret 

Motivation  Identi¬ 
fication 

Incapacitate  Individuals 

S 

S 

P 

S 

S 

The  aggregation  example  applied  the  following  considerations  (resulting  in  the  weights’  vector  shown  in 
Table  3-7): 

•  All  the  primary  BRs  are  equally  weighted. 

•  All  the  secondary  BRs  are  equally  weighted. 

•  The  primary  BRs  are  considered  twice  as  important  as  the  secondary  BRs. 


Table  3-7:  Weights’  Vector  for  Incapacitate  Individuals 

Mobility  Communi-  Physical  Group  Sense  &  Motivation  Identi- 

cations  Function  Cohesion  Interpret  fication 

Incapacitate  Individuals  1/6  1/6  1/3  0  1/6  1/6  0 


In  the  AHP  case,  if  wp  =  2  ws,  then  the  relative  importance  of  the  BRs  to  build  up  the  criteria  comparison 
matrix  is: 


Table  3-8:  Importance  of  BRs  for  Incapacitate  Individuals 


Mobility 

Communi- 

Physical 

Group 

Sense  & 

Motivation 

Identi- 

cations 

Function 

Cohesion 

Interpret 

fication 

Incapacitate  Individuals  4  4  8  1  4  4  1 


Finally,  the  criteria  comparison  matrix  is: 

Table  3-9:  Criteria  Comparison  Matrix  for  Incapacitate  Individuals 


Mobility 

Communi¬ 

cations 

Physical 

Function 

Group 

Cohesion 

Sense  & 
Interpret 

Motivation 

Identi¬ 

fication 

Mobility 

1 

1 

*/2 

4 

1 

1 

4 

Communications 

1 

1 

Vi 

4 

1 

1 

4 

Physical  Function 

2 

2 

1 

8 

2 

2 

8 

Group  Cohesion 

1/4 

1/4 

1/8 

1 

1/4 

1/4 

1 

Sense  &  Interpret 

1 

1 

Vi 

4 

1 

1 

4 

Motivation 

1 

1 

Vi 

4 

1 

1 

4 

Identification 

1/4 

1/4 

1/8 

1 

1/4 

1/4 

1 
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Table  3-10:  Methods’  Results  for  Incapacitate  Individuals 


Non  User  Weighted  Methods 


User  Weighted  Methods 


p\ 

SAS-035 

RMS 

RWG 
(N  =  25) 

PCW 

WED 

SM 

AHP 

ANN 

NLW  1 :  Optical  Disrupter 

1.0000 

0.6372 

0.0868 

0.0750 

0.6432 

0.5250 

0.3035 

0.4906 

NLW  5:  Kinetic  (Blinis) 

0.9992 

0.5949 

0.1389 

0 

0.5268 

0.4500 

0.3756 

0.4508 

NLW  6:  Kinetic  (12  gauge) 

0.9844 

0.4614 

0.0192 

0 

0.5492 

0.4500 

0.3209 

0.4400 

If  we  applied  the  aggregation  methods  to  the  “Stop/Interdict  Vehicles,  Vessels,  and  Aircraft”  case  the 
NLWs  used  need  to  be  anti-materiel. 


Table  3-11 :  Linkages  from  Tasks  to  BRs:  Stop/Interdict  Vehicles,  Vessels,  and  Aircraft 


Mobility 

Communi¬ 

cations 

Physical 

Function 

Group 

Cohesion 

Sense  & 
Interpret 

Motivation 

Identi¬ 

fication 

Task  -  Stop/Interdict 
Vehicles,  Vessels,  and 
Aircraft 

P 

P 

S 

Weights 

2/5 

0 

2/5 

0 

1/5 

0 

0 

Importance  (for  AHP) 

8 

1 

8 

1 

4 

1 

1 

And  the  criteria  comparison  matrix: 


Table  3-12:  Criteria  Comparison  Matrix  for  Stop/Interdict  Vehicles,  Vessels,  and  Aircraft 


Communi¬ 

cations 


Physical 

Function 


Group 

Cohesion 


Sense  & 
Interpret 


Motivation 


Identi¬ 

fication 


Mobility 


1 


8 


8 


Communications 

1/8 

1 

1/8 

1 

1/4 

1 

1 

Physical  Function 

1 

8 

1 

8 

2 

8 

8 

Group  Cohesion 

1/8 

1 

1/8 

1 

1/4 

1 

1 

Sense  &  Interpret 

1/2 

4 

1/2 

4 

1 

4 

4 

Motivation 

1/8 

1 

1/8 

|  1 

1/4 

1 

1 

Identification 

1/8 

1 

1/8 

1 

1/4 

1 

1 

The  results  are  presented  in  Table  3-13. 
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Table  3-13:  Methods’  Results  for  Stop/Interdict  Vehicles,  Vessels,  and  Aircraft 


Non  User  Weighted  Methods 

User  Weighted  Methods 

P*1 

SAS-035 

RMS 

RWG 
(N  =  25) 

PCW 

WED 

SM 

AHP 

ANN 

NLW  2:  Vehicle  Stopper 
(30  mph) 

1.0000 

0.5323 

0.3108 

0.7967 

0.8908 

0.7967 

0.4478 

0.7118 

NLW  3:  Vehicle  Stopper 

0.9994 

0.5211 

0.3042 

0.7798 

0.8719 

0.7798 

0.4407 

0.6975 

(49  mph) 


NLW  4:  Vehicle  Stopper 

0 

0 

0 

0 

0 

0 

0.1114 

(70  mph) 

The  most  appropriate  methods  to  take  into  account  linkages  from  tasks  to  BRs  (priorities  among  BRs)  are 
user  weighted  ones  (PCW,  WED,  SM,  AHP,  ANN). 

3.5.2.3  Comparison  of  Methodologies 

Table  3-14  shows  a  comparison  of  the  various  aggregation  methodologies. 

Table  3-14:  Comparison  of  Methodologies 


METHOD 

Analytical 

formula 

User  criteria 

Pairwise 

dominance 

Expert  input 
required? 

Comments 

SAS-035 

Yes 

No 

No 

No 

Saturation:  If  any  P,  =  1 ,  then 
the  aggregated  P*  =  1. 

RMS 

Yes 

No 

No 

No 

An  appropriate  method  when 
priority  among  BRs  is  not 
needed. 

WED 

Yes 

Yes 

No 

Yes 

Assign  relative  importance  to 
the  BRs. 

SM 

No 

Yes 

No 

Yes 

Assign  relative  importance  to 
the  BRs. 

RWG 

No 

No 

Yes 

No 

Weights  determined  by 
uniformly  random  generation. 
Results  are  averaged. 

PCW 

No 

Yes 

Yes 

Yes 

Assign  relative  importance  to 
the  BRs  by  pairwise. 

AHP 

No 

Yes 

Yes 

Yes 

Rank  the  BRs  in  comparison 
to  each  other.  Adequate 
method  when  there  are  few 
NLW  selected.  Accurate  and 
realistic  results. 

ANN 

No 

Supervised 
nets  are  trained 
using  results 
from 
the  other 
methods 

It  compares 
every  NLW 
to  the  others 
(one  at  a 
time,  not  pair 
by  pair) 

Supervised 
nets  Yes 
Unsupervised 
No 

Results  depend  on  the  methods 
used  to  train  the  net. 

Good  choice  when  database  is 
incomplete.  The  best  method 
when  there  are  a  large 
number  of  alternatives. 
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A  software  script  was  used  to  compare  the  different  methods’  results  across  a  range  of  randomly  generated 
MoSE  values.  In  the  comparative  analysis,  the  inputs  (1000  NLW)  are  randomized  (effectiveness  values 
of  the  7  BRs  uniformly  distributed  between  0  and  1,  with  BRs  weighted  equally)  and  the  variability  of 
effectiveness  values  is  simulated  by  means  of  a  Gaussian  random  generator  (100  simulations  for  each 
NLW).  The  Monte  Carlo  method  is  then  applied  running  100,000  simulations  for  each  aggregation 
method4.  Conclusions  about  certain  limits  of  use  of  the  methods  are  presented.  However,  experimental 
data  is  absolutely  required  to  decide  which  one  of  the  methods  is  the  most  appropriate  to  use  in  MoSE 
aggregation. 

The  resulting  matrix  of  correlations  between  the  scores  obtained  from  the  different  methods  is  as  follows: 


Table  3-15:  Correlation  Matrix  (r) 


SAS-035 

RMS 

WED 

SM 

RWG 

PCW 

AHP 

ANN 

SAS-035 

1 

0.6634 

0.6634 

0.6007 

0.2869 

0.2358 

0.3908 

0.6118 

RMS 

0.6634 

1 

1 

0.9665 

0.6390 

0.5958 

0.5860 

0.9457 

WED 

0.6634 

1 

1 

0.9665 

0.6390 

0.5958 

0.5860 

0.9457 

SM 

0.6007 

0.9665 

0.9665 

1 

0.6583 

0.6245 

0.6612 

0.9714 

RWG 

0.2869 

0.6390 

0.6390 

0.6583 

1 

0.9722 

0.8326 

0.7704 

PCW 

0.2358 

0.5958 

0.5958 

0.6245 

0.9722 

1 

0.7917 

0.7276 

AHP 

0.3908 

0.5860 

0.5860 

0.6612 

0.8326 

0.7917 

1 

0.8069 

ANN 

0.6118 

0.9457 

0.9457 

0.9714 

0.7704 

0.7276 

0.8069 

1 

If  a  high  correlation  between  methods  is  obtained  (r  greater  than  0.7),  then  the  methods  follow  the  same 
“tendency”.  However,  high  correlation  between  methods  does  not  prove  the  two  methods  agree  and  the 
obtained  values  cannot  be  considered  accurate  or  adequate  for  our  interests.  Nevertheless,  this  would  be  a 
good  starting  point  to  begin  the  comparison  of  methods. 

The  proposed  SAS-035  method  has  lower  correlations  with  all  the  other  methods.  Moreover,  the  saturation 
issue  previously  identified  may  limit  the  method’s  usefulness. 

Since  all  the  weights  of  the  BRs  are  equal  (for  this  example  case),  the  aggregation  formulas  are  the  same 
and  there  is  a  perfect  correlation  between  the  RMS  and  WED  methods  (r  =  1). 

The  SM  shows  a  high  correlation  with  the  RMS,  the  WED  and  the  ANN,  but  the  correlation  presented 
with  the  RWG  and  the  PCW  is  poor.  The  last  two  methods  have  good  correlations  between  one  another 
and  poor  with  the  rest  of  the  methods. 

The  AHP  and  the  ANN  have  high  correlation  between  them.  However,  the  ANN  presents  higher 
correlations  with  the  other  methods. 

Figure  3-4  shows  the  correlation  averages  for  the  different  methods: 


4  Examples  of  10  inputs  generated  are  presented  in  Annex  A.3.  User  criteria  give  the  same  relative  importance  to  all  BRs  when 
it  is  needed. 


RTO-TR-SAS-060 


3-17 


AGGREGATING  RESULTS  FOR 

SYSTEM  AND  OPERATIONAL  EFFECTIVENESS 


ORGANIZATION 


SAS035  RMS  WED  SM  RWG  PCW  AHP  ANN 

Figure  3-4:  Average  Correlations  for  Each  Method. 


ANN  has  the  highest  correlations,  with  the  SM,  RMS  and  WED  also  having  high  correlations,  whereas  the 
SAS-035  has  the  lowest  as  mentioned.  Even  if  the  highest  correlations  do  not  mean  the  best  methods,  it  is 
recommended  to  get  a  high  one. 

Next,  a  comparison  of  methods  is  made  plotting  the  scatter  diagrams  of  the  data  obtained  by  the  Monte 
Carlo  method  (Figure  3-5).  Scatter  diagram  give  a  valuable  insight  into  the  way  the  different  methods  are 
related.  In  Figure  3-5,  the  values  of  a  method  are  plotted  against  the  corresponding  values  of  the  other 
methods. 
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Figure  3-5:  Scatter  Diagrams  of  the  Methods. 
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All  the  methods  studied  present  a  positive  slope,  which  means  that  the  aggregation  values  of  the  methods 
are  broadly  in  concordance.  If  the  correlation  is  perfect,  the  scatter  diagram  becomes  a  straight  line. 

As  seen  in  the  scatter  diagrams,  certain  methods  reach  the  axis,  which  are  the  bounds  of  the  values.  This 
happens  with  the  SAS-035,  RWG,  and  PCW  methods.  These  methods,  which  have  values  on  the  limits, 
could  be  considered  inappropriate  for  aggregation. 

Of  the  other  methods,  four  (WED,  SM,  AHP  and  ANN)  take  into  account  user  criteria,  and  one  (RMS) 
does  not.  When  aggregating  effectiveness  values,  user  criteria  are  desirable.  Therefore,  the  RMS  could  be 
eliminated  from  the  list  of  methods.  Moreover,  if  relative  importance  is  equal  for  all  the  BRs,  the  RMS 
and  WED  formulations  are  identical. 

The  four  remaining  methods  are:  WED,  SM,  AHP  and  ANN.  Figure  3-5  graphs  with  AHP  show  a  wider 
and  less  clear  cloud  of  points.  However,  this  method  includes  a  process  for  determining  weights. 

The  mathematical  relation  between  methods  may  be  estimated  by  a  regression  line.  The  disagreement 
between  methods  is  measured  by  the  departure  of  the  regression  line  from  the  bisecting  line  of  the  plot 
(unitary  slope)  and  the  offset  (the  intersection  with  the  y-axis). 

Regression  lines  have  been  calculated  for  the  methods  and  two  indexes  of  merit  are  obtained  (see  Annex 
D):  absolute  values  of  deviation  from  unitary  slope  and  offset.  These  indices  are  averaged  for  each  method 
and  represented  in  Figure  3-6.  If  the  values  of  the  indexes  averaged  are  lower,  then  the  agreement  between 
one  method  and  the  remaining  three  is  higher.  Consequently,  ANN,  WED  and  SM  show  higher  concordance 
than  the  AHP. 


0,7 
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0,5 

0,4 

0,3 

0,2 
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ANN  WED  SM  AHP 


Figure  3-6:  Regression  Lines  Indexes  of  Merit. 


3.6  VARIABILITY  ANALYSIS  AND  UNCERTAINTY  TREATMENT 

3.6.1  Definitions 

The  definitions  of  uncertainty  and  variability  are  included  below: 
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•  Uncertainty:  Lack  of  knowledge  about  specific  factors,  parameters,  or  models5. 

•  Variability:  Refers  to  observed  differences  attributable  to  true  heterogeneity  or  diversity  in  a 
population  or  exposure  parameter  (natural  random  processes).  Taken  into  account  as  population 
distribution  model  (univariate  normal  distribution  for  each  BR)6. 

3.6.2  Variability  Analysis:  Monte  Carlo  (MC)  Simulations 

The  aggregation  methods  (except  RWG)  are  deterministic;  this  means  that  if  the  inputs  are  defined, 
then  the  aggregation  value  obtained  is  the  same  every  time  the  method  is  calculated  (results  are  independent 
of  how  many  times  aggregation  is  recalculated  -  i.e.,  number  of  simulations  or  iterations). 


INPUTS 


MODEL 


OUTPUTS 


P  (Pl,P2,P3,P4,P5,P6,Py) 


DETERMINISTIC 

MODEL 


Aggregated 

Value 


INPUTS 


MODEL 


OUTPUTS 


P  (P;  Probability  Density  Function)  STOCHASTIC 

MODEL 


P*  Aggregated 
Value 


Figure  3-7:  Monte  Carlo  Simulations  Method. 

The  Monte  Carlo  Method  was  applied  in  order  to  estimate  the  impact  of  MoSE  on  aggregated  results  for 
the  different  aggregation  methods.  The  Monte  Carlo  process  steps  are  as  follows: 

1)  Define  the  Problem/System  definition. 

2)  Generate  pseudo-random  numbers. 

3)  Generate  pseudo-random  variables. 

4)  Simulate  N-times  applying  a  model  or  method  (i.e.,  aggregation  method). 

5)  Statistical  Analysis  of  the  N  simulations  results. 

The  pseudo-random  number  generation  used  produces  Gaussian  random  values  by  using  the  Ziggurat 
method.  The  pseudo-random  numbers  and  variables  generation  are  shown  in  Annex  D. 

Given  an  input  of  7  effectiveness  values,  the  initial  model  proposed,  is  based  on  7  normal  distribution 
functions  with  the  following  hypothesis: 

•  Mean  value  of  the  effectiveness  based  on  each  BR:  pi  =  Pi  (i  =  1,  . .  .7) 

•  Standard  deviation:  ai  proportional  to  P;  (i  =  1,  . .  .7) 

That  is  to  say,  the  standard  deviation  a;  proportional  to  the  effectiveness  value  for  each  BR  and  the  mean 
Pi  is  the  corresponding  value.  An  input  of  a  semi-empirical  parameter  e  must  be  given  by  the  user,  so  a; 
can  be  calculated: 

5  More  advanced  models  may  include:  Measurement  and  systematic  errors;  Due  to  the  simplification  of  the  real  world. 

6  More  advanced  models  may  include:  Effects;  Physiological  variability. 
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I  3  cti  =  e  Pj  I 

Figure  3-8:  Normal  Distribution  of  Each  P|. 


Using  the  proposed  model,  Figure  3-9  shows  one  example  of  the  generation  of  the  pseudo-random 
variables  is  represented  given  the  semi-empirical  parameter  e  =  0. 1  (10%)  and  the  following  vector  P 
(0.1,  0.15,  0.2,  0.3,  0.5,  0.6,  0.8)* 7.  The  number  of  simulations  for  the  example  is  N  =  1000. 


0.4  0.5  0.6 

P  -BR  values- 


Figure  3-9:  Normal  Probability  Density  Functions  and 
Monte  Carlo  Simulations  to  Cope  with  Variability. 


This  Monte  Carlo  model  ensures  Pi  lies  between  0  and  1  as  follows8: 

•  if  Pi  <  0,  then  run  new  simulation  until  Pi  >  0 

•  if  Pi  >  1 ,  then  run  new  simulation  until  P;  <  1 

As  a  result  of  these  boundary  conditions,  the  probability  density  functions  of  the  P;  cease  to  be  normal. 
Statistical  Analysis  of  the  results  has  been  carried  out,  obtaining  the  mean  (which  is  a  new  aggregated  value) 
and  the  standard  deviation. 


*7 

Not  based  on  real  data. 

8  Boundaries  might  be  between  -1  and  1. 
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The  pairwise  comparison  methods  (AHP  and  PCW)  cannot  be  applied  directly  to  cope  with  variability. 
Next,  an  approach  of  the  Analytic  Hierarchy  Process  combined  with  Monte  Carlo  is  presented. 

The  AHP  attaches  weights  to  criteria  and  determines  how  well  each  alternative  scores.  If  variability  is 
ignored,  the  scores  are  unique  (single- valued).  The  AHP  aggregation  method  can  be  extended  with 
information  about  variability  by  combining  it  with  the  Monte  Carlo  method,  which  takes  the  range  of 
possible  values  of  the  variables  and  their  probability  into  account.  This  combination  derives  in  a  probabilistic 
score  for  each  NLW  option  on  the  various  criteria.  It  allowed  us  to  evaluate  differences  in  scores  but  also  a 
possible  overlap  between  them.  Figure  3-10  illustrates  how  the  AF1P  and  Monte  Carlo  approaches  are 
combined  (Annex  D  provides  additional  detail). 


AIIP 

Comparison  of  alternatives  by  a  number  of 
conflicting  criteria  with  a  certain  weight. 
For  each  criterion,  a  unique  score  has  to  be 
determined 
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X 

65 

Muntc  (  a Ho  method 

Simulation  of  stochastic  variables,  eg 
with  a  normal  distribution  For  each  trial,  a 
value  is  chosen  from  the  distribution  and 
an  output  is  calculated 

Result . 
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sum  of  all  weights  times  scores. 

5  =  y  w  .s  ' 

a  ^  c  a 
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Mean  value  of  the  overall  score  and 
distinction  between  alternatives  is 
important. 


Figure  3-10:  Combination  of  AHP  and  Monte  Carlo  Approaches  in  a  New  Method9. 


The  next  step  is  to  apply  the  aggregation  methods10.  Table  3-16  shows  results  obtained  by  the  WED,  SM 
and  AHP  methods,  with  e  =  0. 1  and  N  =  1000  simulations. 


g 

“Comparing  uncertain  alternatives  for  a  possible  airport  island  location  in  the  North  Sea  Ocean  &  Coastal  Management”. 
Caroline  S.  van  der  Kleij,  Suzanne  J.M.H.  Hulscher,  Teunis  Louters,  2003. 

111  The  inputs  (Pp  from  Table  1-1  are  considered. 
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Table  3-16:  Results  Obtained  by  WED,  SM  and  AHP  and  its  MC  Combinations 


NLW 

WED 

WED  &  MC 

SM 

SM  &  MC 

AHP 

AHP  &  MC 

Mean  (p) 

Dev  (a) 

Mean  (p) 

Dev  (a) 

Mean  (p) 

Dev  (n) 

Optical  Disrupter 

0.6372 

0.6345 

0.0088 

0.4786 

0.4771 

0.0071 

0.1933 

0.1934 

0.0031 

Vehicle  Stopper 
(30  mph) 

0.5323 

0.5261 

0.0078 

0.2846 

0.2812 

0.0043 

0.0958 

0.0959 

0.0019 

Vehicle  Stopper 
(49  mph) 

0.5211 

0.5197 

0.0097 

0.2786 

0.2775 

0.0055 

0.0943 

0.0944 

0.0018 

Vehicle  Stopper 
(70  mph) 

0 

0 

0 

0 

0 

0 

0.0247 

0.0247 

0.0008 

Kinetic  (Blinis) 

0.5949 

0.5945 

0.0096 

0.5072 

0.5071 

0.0072 

0.3424 

0.3423 

0.0051 

Kinetic  (12  gauge) 

0.4614 

0.4615 

0.0079 

0.3857 

0.3859 

0.0058 

0.2496 

0.2496 

0.0036 

The  mean  from  combining  the  two  methods  is  approximately  equal  to  the  results  obtained  by  the  simple 
methods.  However,  the  combination  provides  a  standard  deviation  that  quantifies  the  variability  of  the 
effectiveness  measures.  Finally,  an  example  of  the  representation  of  the  Optical  Disrupter  (WED-MC) 
is  presented: 


MONTE  CARLO  SIMULATIONS 

Figure  3-1 1 :  WED-MC  Variability  Representation  for  the  Optical  Disrupter. 

3.6.3  Uncertainty  Treatment:  Fuzzy  Logic  (FL) 

Traditional  MCDMs  require  precise  input  data  to  determine  the  preferred  alternative  based  on  the 
performance  characteristics  of  each  alternative.  Unfortunately,  only  a  limited  amount  of  operational  data 
exists  for  NLW  Effectiveness  assessment.  Moreover,  the  limited  available  data  is  dependent  on  the 
operational  conditions,  and  is  characterized  by  uncertainty,  subjectivity,  imprecision  and  ambiguity.  Fuzzy 
sets  theory  provides  a  mathematical  means  to  order  alternatives  according  to  preference  using  imprecise, 
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subjective,  and  uncertain  performance  characteristics  and  subjective  and  ambiguous  stakeholder  opinions 
and  preferences. 

The  block  diagram  of  the  fuzzy  logic  application  is  the  following: 


-  Extremely  important 


Figure  3-12:  Fuzzy  Logic  Application. 

A  statement  of  the  Fuzzy  Logic  Theory  is  provided  in  Annex  D.  The  following  illustrative  example 
demonstrates  the  application  of  fuzzy  set  concepts.  Six  alternatives  will  be  evaluated  using  the  seven 
criteria  (BRs).  User  preferences  will  also  be  considered.  The  objective  is  to  rank  alternatives. 

The  analytical  procedure  follows  the  following  six  steps: 

1)  Determine  the  alternatives  and  relevant  criteria  to  be  evaluated.  In  this  case,  the  alternatives  are 
the  NLW  from  Table  3-4,  and  the  criteria  are  the  BRs. 

2)  Convert  alternative  performance  measures  (P;)  to  fuzzy  numbers.  The  uncertainty  in  the  measures  is 
10%  of  each  MoSE.  (The  new  input  matrices  -  each  NLW’s  P|  for  Red  -  are  shown  in  Annex  D.) 

3)  Generate  stakeholder  preference-based  criteria  weights.  The  criteria  become  triangular  fuzzy 
numbers  predefined  linguistically  by  the  fuzzy  logic  theory.  In  this  case,  all  BRs  are  of  equal 
importance. 

4)  Apply  the  corresponding  method  for  the  three  different  matrices  (lower,  mean  and  upper) 
independently. 

5)  Rank  the  alternatives  using  multi-attribute  decision  making  methods. 

6)  Finally,  use  the  Kaufmann  and  Gupta’s  method  to  rank  the  fuzzy  sets. 


Figure  3-13:  Stages  of  the  Fuzzy  Application. 


Experience  has  shown  that  fuzzy  logic  allows  decision  making  with  estimated  values  in  spite  of  incomplete 
information.  However  it  should  be  noted  that  a  decision  may  not  be  correct  and  can  be  improved  later  when 


RTO-TR-SAS-060 


3-25 


AGGREGATING  RESULTS  FOR 

SYSTEM  AND  OPERATIONAL  EFFECTIVENESS 


ORGANIZATION 


additional  information  is  available.  Of  course,  a  complete  lack  of  information  will  not  support  any  decision 
making  using  any  form  of  logic. 

Fuzzy  logic  may  be  applied  to  Neural  Networks  becoming  a  Fuzzy-Neural  hybrid  system  that  is  an 
efficient  method.  The  work  essentially  consists  of  two  stages:  a  Fuzzy  system  and  an  Artificial  Neural 
Network  system.  The  initial  stage  accounts  for  the  uncertainty  due  to  imprecise  effectiveness  data. 
The  final  stage  is  comprised  of  classification  of  the  BRs  based  on  the  outputs  obtained  from  the  trained 
neural  network.  Radial  Basis  Function  (RBF)  neural  networks  were  applied  to  fuzzy  systems. 

The  rankings  obtained  for  the  different  methods  are  presented  in  Table  3-17. 

Table  3-17:  Ranking  of  Fuzzy  Combined  with  WED,  SM,  AHP  and  ANN 


NLW 

Ranking 

Fuzzy  WED 

Fuzzy  SM 

Fuzzy  AHP 

Fuzzy  ANN 

Optical  Disrupter 

1 

2 

3 

2 

Vehicle  Stopper  (30  mph) 

3 

4 

4 

4 

Vehicle  Stopper  (49  mph) 

4 

5 

5 

5 

Vehicle  Stopper  (70  mph) 

6 

6 

6 

6 

Kinetic  (Blinis) 

2 

1 

1 

1 

Kinetic  (12  gauge) 

5 

3 

2 

3 

3.7  AGGREGATION  THROUGH  PHASES  (OPTIMIZING  THE  USE  OF 
AVAILABLE  TYPES  OF  NLWs  THROUGH  THE  DIFFERENT  PHASES 
OF  A  MISSION) 

In  this  section,  an  optimization  example  of  the  use  of  available  types  of  NLWs  through  the  different 
phases  of  a  mission  is  presented.  It  is  assumed  that  one  type  of  NLW  is  used  in  each  phase  (see  Annex  D). 
Certain  definitions  and/or  the  user  criteria  are  needed  to  specify  the  phases’  characteristics.  These  are: 

•  Operational  Evaluation:  This  is  important  to  quantify  the  relative  importance  of  phases. 

The  proposal  is  to  parameterize  this  input  according  to  the  following  criteria: 


Table  3-18:  Operational  Evaluation  Parameterization 


Importance  of  the  Phase 

Operational 

Evaluation 

Importance  of 
the  Phase 

Operational 

Evaluation 

Null/Unnecessary 

1  -  10 

Essential 

51-60 

Insignificant 

11-20 

Big 

61-70 

Very  Small 

21-30 

Enormous 

71-80 

Small 

31-40 

Extreme 

81-90 

Near  Essential 

41-50 

Super  Extreme 

91-100 
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•  Duration-.  It  is  imperative  to  know  the  time  duration  of  each  phase.  Without  this  information,  it  is 
not  possible  to  calculate  accurate  results. 

•  Relative  importance-.  At  times  it  is  useful  to  weight  the  importance  of  accomplishing  the  task  (Pi) 
relative  to  associated  constraints  (P2).  This  is  only  meaningful  for  Red.  Blue  and  white  colours 
only  have  P2. 

•  Consumption  of  ammunition  per  time  unit  per  each  NLW  selected. 

•  Maximum  availability  of  ammunition  per  each  NLW  for  all  the  phases. 

After  defining  all  the  characteristics  for  the  different  phases,  the  aggregation  through  phases  can  be 
undertaken. 

Example:  USE  OF  THE  ALGORITHM  TO  OPTIMIZE  THE  USE  OF  NLWS  THROUGH  PHASES 
(NLWEESS  APPLICATION) 

This  example  shows  produced  by  applying  the  NLWEESS  software.  This  example  involves  the  protection 
of  a  military  compound.  The  details  of  the  scenario  are  in  Annex  D. 

3.7.1  Inputs 

Several  types  of  NLW  are  used  to  defend  the  compound,  which  are  distributed  among  the  10  personnel 
appointed  to  the  compound  protection  (control  force)  as  shown  in  Table  3-19. 

Table  3-19:  Available  NLW 


NLW 

Technology 

Distribution 
(Soldiers  Carrying 
The  NLW) 

Unit 

Consumption 
(Per  Min.) 

Total 

Consumption 
(Per  Min.) 

Number 

Maximum 

(Available 

Ammunition) 

Optical 

Disrupter 

Optic-Electronic 

(Laser) 

5 

1 

5 

** 

Kinetic 

Kinetic  (Impact) 

5 

8* 

40 

450 

Malodorant 

(hand 

thrown) 

Chemical 

(Irritant) 

5 

1 

5 

20 

Llash-Bang 

(hand 

thrown) 

Acoustic-Optic 

5 

1 

5 

20 

HPM 

Electromagnetic 

1 

1 

1 

** 

Vehicle 

Stopper 

Confinement 

(Nets) 

3 

1 

3 

15 

*  Time  of  charge  included. 

**  A  near-infinite  supply  was  assumed  when  evaluating  directed  energy  (optical  disrupter  and  HPM). 


The  duration  and  total  NLW  consumption  per  phase  (if  allowed  by  NLW  availability)  is  shown  in 
Table  3-20.  Obviously,  NLW  availability  may  constrain  total  consumption. 
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Table  3-20:  Duration  and  Total  NLW  Consumption  Per  Phase 


Phase  0 

Phase  1 

Phase  2 

Phase  3 

Duration  of  the  Phase  (min) 

10 

15 

15 

10 

Optical  Disrupter 

- 

75 

75 

50 

Kinetic 

- 

600 

600 

400 

Malodorant 

- 

75 

75 

50 

Flash-Bang 

- 

75 

75 

50 

HPM 

- 

15 

15 

10 

Vehicle  Stopper 

— 

45 

45 

30 

The  operational  evaluation  (weighting  the  relative  importance  of  phases)  is  shown  in  the  following  table: 

Table  3-21:  Operational  Evaluation  Per  Phase 


Phase  0  Phase  1 


Phase  2 


Phase  3 


Operational  Evaluation1 


100 


50 


100 


In  the  above  table,  please  note  that  the  phases  considered  more  important  are: 

•  Phase  1,  because  eliminating  the  threat  before  it  becomes  a  larger  problem  is  preferred. 

•  Phase  3,  because  the  threat  must  be  eliminated  completely  before  using  lethal  weapons,  anticipating 
an  increase  of  violence. 

The  relative  importance  of  task  accomplishment  (Pi)  versus  constraint  satisfaction  (P2)  is  given  as  the 
parameter  A.  For  this  example,  the  values  by  phase  are: 

Table  3-22:  Relative  Importance  of  Pi  and  P2  Per  Phase 


A,  (Pi  vs.  1-P2)  3  0.0  0.5  1 


Phase  0 

Phase  1 

Phase  2 

Phase  3 

Reversibility  is  more  important  (see  constraints  of  the  Phase  1)  when  A  is  0.  As  the  sequence  of  phases 
proceeds,  reversibility  is  less  significant.  Finally  in  Phase  3,  the  effects  produced  on  the  target  are  essential 
and  reversibility  is  not  deemed  important  (see  constraints  of  the  Phase  3),  so  A  is  1. 

3.7.2  Results 

The  aggregation  method  chosen  for  this  example  is  AHP.  In  this  case,  the  most  important  BRs  are  Mobility 
and  Motivation  (both  with  a  rating  of  7),  and  the  other  BRs  have  an  importance  rating  of  1,  yielding  the 
following  matrix: 


ii 


The  Operational  Evaluation  is  estimated  from  0  to  100,  following  the  relationship  quali-quantitative  presented  on  page  3-26. 
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Table  3-23:  Matrix  of  Relative  Importance  for  the  AHP 

Mobil. 

Commun¬ 

ication 

Physical 

Function 

Sense  & 
Interpret 

Group 

Cohesion 

Identif¬ 

ication 

Motiv¬ 

ation 

Mobility 

1 

7 

7 

7 

7 

7 

1 

Communication 

0.1429 

1 

1 

1 

1 

1 

0.1429 

Physical  Function 

0.1429 

1 

1 

1 

1 

1 

0.1429 

Sense  &  Interpret 

0.1429 

1 

1 

1 

1 

1 

0.1429 

Group  Cohesion 

0.1429 

1 

1 

1 

1 

1 

0.1429 

Identification 

0.1429 

1 

1 

1 

1 

1 

0.1429 

Motivation 

1 

7 

7 

7 

7 

7 

1 

The  results  are  as  follows: 

Table  3-24:  Results  for  One  Phase  (Red) 


Pi 

P2 

NLW  1:  Kinetic  (Blinis) 

0.2268 

0.3494 

NLW  2:  Optical  Disrupter 

0.1273 

0.2077 

NLW  3:  Vehicle  Stopper  (30  mph) 

0.1178 

0.0827 

NLW  4:  Malodorant 

0.0328 

0.0079 

NLW  5:  Flash-Bang 

0.2088 

0.11 

NLW  6:  HPM 

0.2865 

0.2422 

According  to  the  aggregated  Pi  and  P2  values,  the  NLW  that  produces  the  highest  effect  on  the  target  is  the 
HPM  (highest  P|),  whereas  the  NLW  with  the  best  reversibility  is  the  Malodorant  (smallest  P2). 

The  results  cover  6 3  =  216  (number  NLW  number  phases)  different  combinations.  The  output  file  obtained  from 
the  simulation  with  the  final  results  and  the  specifications  used  is  presented  in  Annex  D.  A  few  combinations 
are  shown  to  extract  some  essential  conclusions  in  the  table  below. 

As  observed  in  the  previous  table,  the  best  combinations  use  NLW  number  1  (Kinetic)  in  Phase  1  and  the 
Optical  Disrupter  (NLW  2)  in  Phase  2.  Table  3-25  shows  results  for  all  the  possible  NLW  combinations  in 
Phase  3. 
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Table  3-25:  Results  of  Combinations 


N° 

comb. 

NLW 

Ph.  1 

%  of 

use 

Time  of 

use 

(min) 

NLW 

Ph.  2 

%  of 

use 

Time  of 

use 

(min) 

NLW 
Ph.  3 

%  of 

use 

Time  of 

use 

(min) 

Max. 

1 

1 

75 

11.25 

2 

100 

15 

2 

100 

10 

125.66 

2 

1 

75 

11.25 

2 

100 

15 

5 

40 

4 

124.79 

3 

1 

75 

11.25 

2 

100 

15 

6 

100 

10 

124.26 

4 

1 

75 

11.25 

2 

100 

15 

3 

50 

5 

123.82 

5 

1 

75 

11.25 

2 

100 

15 

4 

40 

4 

123.38 

6 

1 

75 

11.25 

2 

100 

15 

1 

0 

0 

123.12 

7 

1 

75 

11.25 

4 

26.67 

4 

2 

100 

10 

121.63 

... 

... 

... 

... 

... 

... 

... 

... 

... 

... 

... 

37 

1 

75 

11.25 

1 

0 

0 

1 

0 

0 

117.10 

38 

2 

100 

15 

2 

100 

15 

1 

100 

10 

66.07 

39 

2 

100 

15 

1 

8.33 

1.25 

1 

100 

10 

63.49 

... 

... 

... 

... 

**• 

... 

... 

... 

... 

... 

... 

Combination  number  6  raises  an  issue.  The  use  of  kinetic  weapons  in  Phase  3  shows  a  time  of  use  of  0 
minutes  (because  all  kinetic  rounds  have  already  been  expended).  As  this  also  happens  in  other  combinations 
(see  results  in  Annex  D),  the  results  require  some  filtering.  Introducing  conditional  probability  of  occurrence 
through  the  phases  could  be  a  solution  to  eliminate  combinations  that  aren’t  possible. 

Combinations  with  the  highest  rankings  (normalized  between  0  and  100)  are  shown  in  the  following  graph. 
All  of  these  highest  ranking  combinations  begin  with  the  use  of  Kinetic  devices  (NLW  1).  A  comparison 
between  the  best  option  “122”  and  the  others  can  be  made  easily.  For  example,  the  final  option  shown  “11 1” 
has  a  difference  of  relative  value  (100  -  92.87  =  7.13%).  So,  while  there  are  35  options  with  higher  rankings 
than  “1 1 1”,  there  is  not  a  large  difference  in  relative  value  from  the  best  option  (“122”). 


COMBINATIONS  OF  NLWs 


Figure  3-14:  Relative  Ranking  of  Combinations. 
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3.8  TASK  2  CONCLUSIONS  AND  RECOMMENDATIONS 

The  main  goal  of  this  task  was  the  analysis  and  development  of  different  aggregation  methods  for  the 
system  effectiveness  methodology  developed  by  SAS-035  and  refined  by  SAS-060  (through  the  efforts  of 
the  Task  1  working  group). 

3.8.1  Conclusions 

An  algorithm  to  optimize  the  use  of  available  types  of  NLWs  through  the  different  phases  of  a  mission  has 
been  developed.  The  results  give  the  best  combinations  across  phases  for  different  available  NLW.  Thus, 
the  methodology  can  assist  in  addressing  how  a  force  with  different  NLW  might  best  achieve  its  aims  and 
objectives  in  a  particular  scenario. 

The  aggregation  methods  and  algorithms  to  analyze  the  best  NLW  combination  have  been  implemented  in 
a  software  tool  called  NLWEESS.  Multiple  simulations  with  available  and  simulated  data  have  been 
carried  out  using  the  NLWEESS  tool,  with  selected  results  presented  this  report. 

Each  phase  is  related  to  a  probability.  This  probability  is  based  on  the  sequence  of  phases,  and  the 
likelihood  associated  with  the  scenario  ending  in  a  particular  phase  or  continuing  on  to  the  next  phase, 
a  tree  branch  of  probabilities.  The  probability  of  going  from  one  phase  to  the  next  is  not  a  strict  function  of 
the  data  but  must  also  be  informed  by  scenario  and  mission  analysis  (to  include  simulation). 

One  important  point  to  emphasize  is  that  more  real  data  would  be  useful  to  validate  the  aggregation 
process.  Much  more  data  would  be  needed  to  support  statistically  significant  test  samples  and  stronger 
conclusions.  The  analysis  presented  in  this  report  has  been  done  mostly  with  pseudorandom  figures  - 
so  the  results  aren’t  well  suited  for  direct  application,  but  they  have  been  effective  for  testing  the  methods. 

3.8.2  Recommendations 

Environmental  factors  could  be  included  in  the  calculation  of  aggregated  values  to  get  more  realistic 
results  according  to  the  specific  situation.  There  has  been  some  discussion  about  how  environmental 
conditions  may  influence  results.  For  instance,  a  flash  effect  is  much  greater  at  night  than  in  the  day 
(and  much  greater  if  the  optics/eyes  are  unprotected).  Some  proposals  for  addressing  environmental 
factors  are  in  Annex  D,  but  further  research  will  be  needed  to  solve  this  problem. 

The  NLWWEES  software  can  support  testing  of  aggregation  methods,  and  continued  efforts  should  be 
pursued  to  the  extent  that  data  will  support. 

Finally,  a  NLW  wargame  might  have  significant  value,  with  objectives  to: 

•  Agree  on  appropriate  scenarios  and  missions; 

•  Identify  the  key  mission  tasks  to  be  undertaken  within  each  scenario; 

•  Identify  the  requirements  derived  from  these  tasks; 

•  Identify  current  NLW  capabilities  and  capability  gaps; 

•  Identify  the  concepts  that  were  most  usefully  employed; 

•  Identify  procedural  changes  that  were  more  appropriate  in  light  of  the  deployment  of  these  concepts; 
and 

•  Provide  input  for  NATO  and  national  defence  planning  processes. 
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Wargame  issues  and  opportunities  could  include: 

•  Doctrinal  considerations; 

•  Organizational  options; 

•  Training;  and 

•  Analytical  support  to  the  assessment  of  new  NLW,  new  technologies,  systems  and  concepts. 

The  intended  outputs  would  be  a  better  understanding  of: 

•  How  a  force  using  different  NLW  might  best  achieve  its  aims  and  objectives; 

•  The  strengths  and  weaknesses  of  different  NLW;  and 

•  Doctrinal  strengths  and  weaknesses. 
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4.1  INTRODUCTION 

The  Task  3  working  group  had  two  main  objectives: 

1)  Enhance  NLW  awareness;  and 

2)  Co-ordinate  with  other  NATO  NLW  efforts. 

Associated  with  the  first  objective,  the  working  group  identified  supporting  items  such  as  developing  and 
disseminating  relevant  NLW  information  materials  (information  sheets,  briefings,  etc.)  to  interested  parties. 
SAS-060  provided  this  information  to  other  NATO  organisations.  Individual  members  provided  appropriate 
information  to  their  respective  nations,  with  some  multinational  efforts  also  undertaken,  to  include  the 
use  of  existing  NLW  Information/Data  Exchange  Agreements  (IEAs/DEAs)  and  the  development  of  new 
agreements. 

The  second  objective  involved: 

•  The  actual  conduct  of  briefings;  and 

•  Interactions  with  other  NATO  organisations  (with  most  activity  being  with  Allied  Command 
Transformation,  NAAG  Quick  Reaction  Team,  and  NATO  Defence  College). 


4.2  ACTIONS  TO  ENHANCE  NLW  AWARENESS 

During  the  course  of  SAS-060,  the  Task  3  working  group  prepared: 

•  Briefings: 

•  NATO  and  NLW,  NLW  Effectiveness,  and  SAS-060  Overviews  (based  on  the  SAS-035  Linal 
Report  and  ongoing  work  of  SAS-060)  presented  on  multiple  occasions,  including  to  Allied 
Command  Transformation  and  as  a  stage  setting  presentation  at  the  NAAG  Quick  Reaction 
Team’s  kickoff  meeting. 

•  CNAD  NLW  Exhibition  Presentation  offered  in  continuous  display  mode  throughout  the 
exhibition. 

•  National  NLW  Capabilities  Briefings  presented  to  Allied  Command  Transformation. 

•  Information  materials: 

•  Trifold  Brochures  (prepared  by  the  working  group,  with  graphics  and  printing  support 
provided  by  the  Norwegian  Defence  Research  Establishment)  that  have  been  distributed 
widely  among  NATO  organisations  and  member  nations. 

•  Summary  Papers  prepared  and  disseminated  in  conjunction  with  the  CNAD  NLW  Exhibition 
(most  of  these  related  to  NLW,  and  in  particular  to  SAS-060’s  ongoing  work;  the  Task  3 
working  group  also  disseminated,  however,  an  RTO  summary  at  the  request  of,  and  provided 
by,  the  SAS  Panel  Executive). 

•  Posters  displayed  during  the  CNAD  NLW  Exhibition. 

•  NLW  Survey,  soliciting  inputs  for  use  by  SAS-060. 
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•  Proposed  Workshop  Outline  for  NATO  Defence  College,  discussed  in  greater  detail  in  the  next 
section. 

•  NLW  White  Paper  prepared  for  NATO  Defence  College  and  Allied  Command  Transformation, 
discussed  in  greater  detail  in  the  next  section. 


4.3  ACTIONS  TO  CO-ORDINATE  WITH  OTHER  NATO  NLW  EFFORTS 

Task  3  working  group  actions  to  co-ordinate  with  other  NATO  NLW  efforts  obviously  and  intentionally 
had  overlaps  with  the  actions  to  enhance  NLW  awareness.  The  identified  briefings  were  given  in  person  to 
Allied  Command  Transformation,  NAAG  Quick  Reaction  Team,  NATO  Modelling  and  Simulation 
Group,  SAS  Panel,  CNAD  NLW  Exhibition,  and  multiple  Ministries  of  Defence. 

The  greatest  level  of  interaction  and  co-ordination  was  with  ACT,  NAAG  QRT,  and  NATO  Defence 
College. 

ACT  hosted  one  of  the  SAS-060  Study  Team  meetings,  and  one  day  of  this  meeting  was  devoted  to  an 
exchange  of  briefings.  SAS-060  presented  background  information  on  NATO  and  NLW,  NLW  technologies 
and  capabilities  (including  detailed  presentations  and  videos  offered  by  several  SAS-060  member  nations), 
the  work  of  SAS-035  and  SAS-060,  and  issues  and  challenges  (including  the  near  total  gap,  outside  of  RTO 
studies,  in  implementing  the  initial  NATO  NLW  Roadmap).  ACT  briefed  SAS-060  on  the  command  and  its 
reorganisation,  transformation  initiatives,  education  and  training  (including  ACT’s  interest  in  forwarding  the 
SAS-060  provided  NLW  information  to  the  Joint  Force  Training  Centre  and  Joint  Warfare  Centre), 
experimentation  process,  and  concept  development  (where  ACT  would  like  to  initiate  work  on  a  NATO 
NLW  Concept,  which  is  one  of  the  reasons  underlying  development  of  the  NLW  White  Paper). 

The  Task  3  working  group  not  only  provided  a  stage  setting  brief  during  the  NAAG  Quick  Reaction 
Team’s  kickoff  meeting  but  also  provided  a  great  deal  of  supporting  information  on  NLW  technology 
types,  weapon  characteristics,  performance  and  effectiveness  measures  (and  some  available  data), 
operational  requirements  and  identified  capability  gaps.  One  of  the  NAAG  QRT’s  recommendations  is 
initiation  of  a  SAS-060  follow-on  study.  The  Task  3  working  group  discussed  this  with  the  full  SAS-060 
Study  Team,  and  this  is  discussed  further  in  the  Conclusions  and  Recommendations  section  (Chapter  5). 

Finally,  the  Task  3  working  group  interacted  extensively  with  the  NATO  Defence  College  (NDC)  on  a 
NEW  Seminar.  This  seminar,  to  be  hosted  in  2008  by  NDC,  has  a  proposed  outline  that  is  included  in 
Annex  E.  In  addition  to  the  workshop  outline,  the  Task  3  working  group  also  prepared  a  White  Paper  to 
convey  NLW  issues  and  opportunities.  This  NLW  White  Paper  is  also  included  in  Annex  E. 
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Chapter  5  -  CONCLUSIONS  AND  RECOMMENDATIONS 

5.1  TASK  1 

The  principal  aim  of  Task  1  was  to  verify  the  MoSE  methodology.  Five  non-lethal  technologies  were 
assessed  and  as  a  result,  the  MoSE  methodology  was  agreed  as  being  verifiable.  Due  to  a  lack  of  data, 
the  methodology  was  not  fully  verified. 

The  study  team  identified  a  number  of  issues  that  require  further  consideration  but  were  satisfied  that  none 
were  so  serious  as  to  pose  a  threat  to  the  viability  of  the  process.  These  constraints  were: 

•  Lack  of  Orthogonality  of  the  Basis  Response  Variables.  This  relates  to  the  risk  of  overlap 
between  different  variables,  leading  to  double-counting  some  of  the  benefits  of  a  given  weapon. 
This  issue  arose  in  the  cases  of  the  vehicle  stopper  -  where  the  estimated  effects  on  Mobility  and 
Physical  Function  measure  exactly  the  same  thing  -  and  the  Optical  Disrupter. 

•  Choice  of  Appropriate  Basis  Response.  A  related  issue  is  the  difficulty  of  selecting  the  right  Basis 
Response  variable  to  measure  the  effect  of  a  weapon.  This  arose  in  the  case  of  the  High  Power 
Microwave  weapon,  where  the  effect  could  equally  well  be  defined  in  terms  of  the  Communication 
or  Sense  and  Interpret  responses.  A  more  orthogonal  and  explicit  definition  of  the  Basis  Responses 
is  a  partial  solution. 

•  Provision  of  Required  Response.  The  RR  specifies  the  commander’s  requirement  for  the 
weapon,  and  should  ideally  come  from  military  sources.  It  was  felt  that  commanders  would  state 
the  target  RR  in  military  terms  (defeat,  seize,  secure,  deny)  whereas  the  methodology  requires  the 
response  in  terms  of  a  combination  of  the  Basic  Responses.  This  process  of  translation  was  not 
examined  in  S AS -060. 

•  Coupled  Effects.  The  possibility  was  identified  that,  where  more  than  one  weapon  was  used  in 
achieving  the  same  task,  the  overall  effect  might  be  more  than  the  sum  of  the  individual  effects. 
This  issue  was  not  assessed  during  the  process  of  verification. 

•  Derivation  of  MoR  from  MoP.  MoP  were  available  for  all  the  weapons  examined  but  it  was 
noted  that  the  only  ones  for  which  complete  MoR  were  available  were  those  -  vehicle  stopper  and 
HPM  -  whose  effect  is  on  equipment  rather  than  people.  Deriving  MoRs  for  human  subjects 
requires  either  a  validated  theory  (not  known  to  exist)  or  experiments,  which  raise  ethical  and 
practical  issues.  The  study  had  doubts  regarding  the  availability  of  MoR  data. 

•  Issues  Related  to  MoSE.  These  issues  include: 

•  MoSE  normalisation:  in  order  to  calculate  the  MoSE  measures  and  normalise  them  to  lie 
within  [-1,1]  the  methodology  requires  specification  of  a  total  time  period  for  the  scenario, 
and  upper  limits  to  the  effects  achieved  by  the  weapon.  If  the  weapon  effect  does  not  return  to 
zero  in  some  reasonable  time  then  (as  demonstrated  by  the  case  of  the  vehicle  stopper) 
the  values  of  P2  and  P3  will  depend  on  the  time  period  selected  as  the  length  of  the  scenario, 
and  an  agreed  method  of  defining  the  scenario  cut  off  point  is  needed. 

•  Setting  the  upper  limit  did  not  in  practice  cause  problems  in  the  exercise,  but  there  are  at  least 
three  issues: 

•  In  the  case  of  Pi,  how  to  account  for  cases  where  the  weapon  has  a  greater  effect  than 
the  specified  upper  limit. 
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•  In  the  case  of  P2,  where  the  aim  is  to  eliminate  the  effect  as  quickly  as  possible  when 
it  is  no  longer  needed,  a  strong  argument  can  be  made  that  excess  effects  should  be 
included;  but  this  could  lead  to  a  P2  <  -1. 

•  In  the  case  of  P3,  there  is  no  obvious  way  of  setting  the  upper  threshold,  and  hence 
measure  collateral  effects.  A  better  approach  would  be  to  repeat  an  analysis  of  Pi  and 
P2  for  Red,  Blue  and  White.  In  this  way  a  series  of  acceptable  effect  magnitudes, 
durations  and  recovery  times  could  be  specified  for  all  actors.  This  removes  the 
arbitrariness  of  the  “upper  magnitude  comparison  limit”  and  makes  all  calculations 
more  logical  and  defensible. 

•  Sensitivity  Analysis.  While  analysis  of  the  Vehicle  stopper  data  demonstrated  the  feasibility  of 
sensitivity  analysis,  the  generic  use  of  sensitivity  analysis  remained  an  issue  as  it  depended  on 
estimates  being  available  of  the  relative  uncertainty  in  the  various  data  -  RR,  MoR  and  the 
thresholds  used  in  calculating  Pi,  P2  and  P3.  The  availability  of  estimates  was  not  examined. 

•  Software  Implementation.  The  SAS-035  report  envisioned  the  implementation  of  the 
methodology  as  a  computer  tool  or  model.  However  the  experience  of  Task  1  would  suggest  that 
no  single  computer  tool  could  meet  the  needs  of  all  weapon  types.  Given  the  ability  to  specify  the 
MoP,  MoR  and  RR  in  agreed  formats,  then  a  calculation  engine  could  be  created  which  would 
perform  the  various  calculations  for  P!  and  P2.  However,  uncertainty  over  the  format  of  data  sets 
and  fields  would  make  the  creation  of  such  a  tool  a  challenge. 

•  Use  of  Combat  Modelling.  The  possibility  was  identified  that  aggregation  methods  of  deriving 
MoOE  from  MoSE  would  not  adequately  capture  the  impact  of  effects  on  individual  targets 
during  the  course  of  a  scenario.  In  this  case,  the  MoSE  measures  could  only  be  integrated 
adequately  into  a  meaningful  high  level  measure  of  operational  success  by  formal  modelling  or 
wargaming  of  the  overall  sequence  of  events. 


5.2  TASK  2 

The  aim  of  Task  2  was  to  extend  the  MoSE  methodology  by  examining  methods  of  aggregation  and  to  assess 
the  feasibility  of  developing  MoOE.  Multiple  aggregation  methods  were  identified  and  implemented  (but  no 
best  one  identified).  In  addition,  the  team  examined  data  variability  and  uncertainty  using  Monte  Carlo 
analysis  and  Fuzzy  Logic,  respectively.  Finally,  a  software  tool  (NLWEESS)  was  developed  by  Spain  that 
assessed  the  algorithm  from  each  method  of  aggregation  in  order  to  determine  P!  and  P2  across  phases  of  a 
single  scenario. 

A  method  of  aggregation  across  phases  of  a  scenario  was  also  developed.  This  method  identifies  the  best 
NLW  combinations  for  use  in  the  scenario  and  offers  insight  into  how  a  force  with  different  NLWs  might 
best  achieve  its  aims  and  objectives. 

Two  challenges  for  the  future  are: 

•  Estimating  the  probability  of  a  scenario  entering  each  phase.  This  probability  is  based  on  the 
sequence  of  phases,  and  the  likelihood  associated  with  the  scenario  ending  in  a  particular  phase  or 
continuing  on  to  the  next  phase,  a  tree  branch  of  probabilities.  The  probability  of  going  from  one 
phase  to  the  next  is  not  just  a  function  of  the  data  but  must  also  be  informed  by  scenario  and 
mission  analysis  (which  may  need  to  include  simulation). 

•  Validation  of  the  aggregation  process.  Not  enough  real  data  was  available  to  conduct  such  a 
validation,  and  the  analysis  has  been  done  mostly  with  pseudorandom  figures  so  the  results  achieved 
are  not  appropriate  for  direct  application. 
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5.3  TASK  3 

The  aim  of  Task  3  was  to  enhance  the  awareness  of  NLW  and  integrate  with  other  NATO  efforts.  This  has 
been  achieved  through  increased  visibility  and  action,  up  to  CNAD  level,  with  major  efforts  including 
support  for  the  CNAD  NLW  Exhibition,  NAAG  Quick  Reaction  Team/Topical  Group  3  and  the  NLW 
Seminar  planned  for  2008  at  the  NATO  Defence  College  in  Rome. 


5.4  SUMMARY 

The  overarching  conclusion  of  this  study  is  that  the  methodology  developed  during  SAS-035  to  measure  the 
effectiveness  of  NLW  is  a  sound  basis  for  further  development  and  has  significant  potential  for  analytical 
and  operational  use.  The  study  has  highlighted  a  number  of  issues  regarding  the  methodology  that  require 
further  investigation,  none  deemed  insurmountable.  There  remains  a  fundamental  issue  regarding  the 
availability  of  suitable  data  which  can  only  be  met  through  experimentation,  trials  and  exercises. 
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ORGANIZATION 


Non-Lethal  Weapons  Effectiveness  Assessment 
Development  and  Verification  Study  (SAS-060) 


I.  ORIGIN 

A.  Background 

NATO  has  identified  non-lethal  weapons  (NLWs)  as  a  critical  additional  capability  needed  in  order  to 
meet  the  demands  of  future  operations.  This  need  has  been  reinforced  in  recent  peace  support  operations 
(as  highlighted  in  conclusions  from  SAS-041/SAS-048)  and  in  anti -terrorism/counter-terrorism  efforts 
(as  discussed  in  MC472).  Although  the  need  for  NLWs  has  been  formally  recognized,  NATO  and  its 
member  nations  still  must  determine  which  non-lethal  capabilities  to  develop  and  field  and  how  best  to 
employ  them.  NATO  has  begun  to  address  these  issues. 

In  April  1994,  CNAD  tasked  the  NATO  research  organisation  to  establish  a  Specialist  Team  on  Non-Lethal 
Weapons.  The  Team  examined  the  feasibility  and  utility  of  NLWs  in  peacekeeping  and  peace  support 
operations,  and  identified  possibilities  for  international  co-operation  in  research  and  development  on 
promising  technologies.  In  1997  the  Council  identified  the  need  for  a  common  NATO  NLW  policy,  and  in 
1999  the  draft  policy  document  drafted  by  the  Non-Lethal  Weapons  Policy  Team  and  the  Council  approved 
this  as  the  official  NATO  NLW  policy.  In  2000  an  Exploratory  Team  developed  a  NATO  NLW  Roadmap  in 
support  of  Defence  Capabilities  Initiative  Item  2  EE(i).  This  NATO  NLW  Roadmap  led  to  the  initiation  of 
three  studies: 

•  NATO  NLW  Effectiveness  Assessment  (SAS-035),  which  developed  Measures  of  Effectiveness 
(MoEs)  appropriate  for  assessing  system  effectiveness. 

•  Non-Lethal  Technologies  and  Future  Peace  Enforcement  Operations  (SAS-040),  which 
investigated  future  requirements  for  NLWs  and  the  types  of  technologies  that  may  meet  these 
requirements. 

•  The  Human  Effects  of  Non-Lethal  Technologies  (HFM-073),  which  is  gathering  existing 
information  on  human  effects  to  support  future  research. 

The  Roadmap  also  called  for  a  follow-on  study  to  continue  addressing  NLW  effectiveness,  with  the  follow- 
on  to  begin  soon  after  SAS-035  completed  its  efforts.  SAS-035  confirmed  the  need  for  the  follow-on  study, 
and  the  SAS  Panel  directed  an  Exploratory  Team  (ET.AM)  to  develop  plans  for  the  follow-on  study. 

The  importance  of  NLWs  has  been  noted  by  SAS-041/048,  MC472,  the  Bi-SC  Long-Term  Capability 
Requirements  Study,  AC/225,  Land  Group  Three  on  Close  Combat  Infantry  Team  of  Experts  on  NLW/ 
MOUT,  Anti-Personnel  Land  Mine  Alternatives  (SAS-023),  and  Land  Group  Two  on  Close  Combat 
Armour.  Moreover,  the  SAS-035  follow-on  study  has  received  written  endorsement  from  senior  operational 
commanders  at  the  flag  officer  level. 

B.  Justification  (Relevance  for  NATO) 

The  challenges  confronting  NATO  are  very  different  from  those  that  prevailed  throughout  the  Cold  War. 
As  seen  in  ongoing  peace  support  operations  in  the  Balkans  and  in  anti-terrorism/counter-terrorism  efforts, 
NATO  is  facing  new  types  of  threats,  undertaking  new  kinds  of  missions,  performing  a  broad  range  of 
military  tasks  within  difficult  operating  environments,  and  conducting  these  tasks  subject  to  a  variety  of 
policy  and  operational  constraints. 
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As  it  pursues  operational  capabilities  and  concepts  consistent  with  21st  Century  demands,  NATO  needs  to: 

•  Expand  the  set  of  tools  available  to  commanders  and  their  forces,  so  they  can  match  the  right  tool 
to  the  opportunity  at  hand. 

•  Exploit  the  precision  offered  by  NLWs  in  terms  of  type  and  magnitude  of  effect,  onset  time, 
duration,  and  recovery/reversibility. 

•  Create  asymmetries  on  the  battlefield  by  developing  and  employing  capabilities  that  our  adversaries 
lack. 

•  Be  able  to  assess  the  effects  of  NLWs  that  may  be  used  against  NATO  forces  in  the  future. 

NLWs  offer  new  capabilities  across  the  spectrum  of  operations  (including  high-intensity  conflict),  and 
NLWs  can  help  meet  future  operational  demands.  Before  commanders  will  be  comfortable  employing 
NLWs,  however,  they  must  understand  their  effectiveness.  SAS-035  developed  a  methodology  for 
assessing  NLW  effectiveness,  and  that  methodology  now  needs  to  be  tested,  verified,  and  extended. 


II.  OBJECTIVES 

A.  Scope 

Verify  and  extend  the  methodology  developed  by  SAS-035. 

B.  Objectives 

The  specific  objectives  of  the  study  are  to: 

1)  Verify  the  Measures  of  System  Effectiveness  (MoSE)  Methodology: 

•  Identify  necessary  data  types  and  experiments  or  experimental  guidelines  for  generating  them. 

•  Make  explicit  the  process  for  describing  required  responses. 

•  Verify  the  methodology,  with  realistic  data  for  actual  system(s)  in  an  operational  context,  and 
provide  worked  example(s). 

2)  Extend  the  MoSE  Methodology: 

•  Address  potential  uses  and  associated  methods  of  aggregation. 

•  Address  confidence  levels,  including  an  examination  of  potential  sources  of  variation  and  how 
variations  propagate  through  the  methodology’s  calculations. 

3)  Explore  development  of  Measures  of  Operational  Effectiveness  (MoOEs): 

•  Assess  the  feasibility  of  extending  the  methodology  to  account  for  operational  effectiveness 
(i.e.,  the  simultaneous  and/or  sequential  use  of  one  or  more  systems  -  non-lethal  and/or  lethal 
-  to  achieve  a  desired  outcome). 

•  Develop  and  explore  concepts  for  assessing  operational  effectiveness. 

4)  Enhance  awareness  of  NLW  work  and  integrate  with  other  NATO  efforts: 

•  Engage  with  organisations  identified  in  the  NATO  NLW  Roadmap. 

•  Conduct  briefings  for  Allied  Command- Operations  and  Allied  Command-Transformation  and 
others  as  appropriate  and  develop  and  conduct  a  seminar/workshop  if  warranted. 
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C.  Deliverables 

Deliverables  will  be  an  extended  and  verified  methodology,  worked  example(s),  experimentation 
guidelines,  seminar(s)/workshop(s),  interim  briefings  on  key  findings,  and  a  Final  Report  with  results  and 
recommendations.  Conclusions  and  recommendations  will  be  briefed  through  the  NATO  chain  of 
command.  Members  are  responsible  for  briefing  their  respective  nations  as  appropriate. 

D.  Duration 

This  will  be  a  3-year  study  beginning  after  formal  approval. 

III.  RESOURCES 

A.  Membership 

Participants  are  likely  to  include  SAS-035  and  Exploratory  Team  participants  -  Canada,  Denmark,  France, 
Germany,  Italy,  the  Netherlands,  Norway,  Spain,  Turkey,  the  United  Kingdom,  the  United  States,  Allied 
Command  Operations,  and  Allied  Command  Transformation  -  and  others  who  have  an  interest  in  NLWs. 
NC3A  participation  is  desired. 

B.  National  and/or  NATO  Resources  Needed 

Each  nation  is  responsible  for  funding  the  participation  of  its  own  team  members.  Wherever  possible, 
the  team  will  draw  on  national  programmes  and  studies.  It  is  the  intention  to  conduct  a  series  of  focused 
working  sessions  wherever  appropriate.  There  will  also  be  significant  work  undertaken  by  small  groups  or 
individual  nations  in  between  working  sessions  in  order  to  complete  study  efforts  and  required  deliverables. 
Individual  nations  will  be  invited  to  host  one  or  more  working  sessions.  In  addition  each  nation  will  be  asked 
to  identify  particular  resources  that  could  support  the  study  to  include  data  and  funding. 

C.  RTA  Resources 

None. 

IV.  SECURITY  CLASSIFICATION  LEVEL 

The  Study  Team  will  determine  whether  efforts  will  be  classified. 


V.  PARTICIPATION  BY  PARTNER  NATIONS 

Openness  to  Partner  Nations  may  be  precluded  by  study  classification. 


VI.  LIAISON 

The  study  will  draw  from  previous  efforts  such  as  SAS-035,  SAS-040  and  SAS-041/048  and  will 
co-ordinate  its  activities  with  related,  ongoing  NATO  and  national  study  efforts,  including  F1FM-073  and 
the  Land  Group  Three  on  Close  Combat  Infantry  Team  of  Experts  on  NLW/MOUT. 
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